Parallel transfers with sftp (call for testing / advice)

[Ben Lindstrom]

OpenSSH does:

      -R num_requests
              Specify how many requests may be outstanding at any one time.
              Increasing this may slightly improve file transfer speed 
but will
              increase memory usage.  The default is 64 outstanding 
requests.

As well as block size:

      -B buffer_size
              Specify the size of the buffer that sftp uses when 
transferring
              files.  Larger buffers require fewer round trips at the 
cost of
              higher memory consumption.  The default is 32768 bytes.


I suspect for this to be taken seriously it will more than likely need 
to have threads dropped in favor of a different method.  As multiple 
times the OpenSSH team has rejected threads as they cause additional 
complexity and security issues.

Ben



Ron Frederick wrote on 5/5/20 9:42 AM:
> On May 5, 2020, at 3:55 AM, Cyril Servant <cyril.servant at gmail.com> wrote:
>> Peter Stuge wrote:
>>> Matthieu Hautreux wrote:
>>>> The change proposed by Cyril in sftp is a very pragmatic approach to
>>>> deal with parallelism at the file transfer level. It leverages the
>>>> already existing sftp protocol and its capability to write/read file
>>>> content at specified offsets. This enables to speed up sftp transfers
>>>> significantly by parallelizing the SSH channels used for large
>>>> transfers. This improvement is performed only by modifying the sftp
>>>> client, which is a very small modification compared to the openssh
>>>> codebase. The modification is not too complicated to review and validate
>>>> (I did it) and does not change the default behavior of the cli.
>>> I think you make a compelling argument. I admit that I haven't
>>> reviewed the patch, even though that is what matters the most.
>> If you want to review the code, here is a direct link to the patch:
>> https://github.com/openssh/openssh-portable/compare/V_8_2_P1...cea-hpc:V_8_2_P1_PSFTP1 <https://github.com/openssh/openssh-portable/compare/V_8_2_P1...cea-hpc:V_8_2_P1_PSFTP1>
> I haven’t reviewed the patch in detail, but one thing that jumped out at me when looking at this was your choice to use multiple channels to implement the parallelism. The SFTP protocol already allows multiple outstanding reads or writes on a single channel, without waiting for the server’s response. I don’t know if the current OpenSSH client takes advantage of that or not, but it seems to me that you’d get the same benefit from that as you would from opening multiple channels with less setup cost (not having to open the additional channels, and not having to do separate file open calls on each channel). It might also make some other things like the progress bar support easier, as all of the responses would be coming back on the same channel.
>
> You can find an example of this form of parallelism (multiple read/write requests outstanding on a single channel) in AsyncSSH’s SFTP implementation. By default, it will allow up to 128 parallel reads or writes of 16 KB each, and you can adjust both the block size and max outstanding requests via “block_size” and “max_requests” arguments in the calls to methods such as get(), put(), and copy() in the asyncssh.SFTPClient class (https://asyncssh.readthedocs.io/en/latest/api.html#sftpclient <https://asyncssh.readthedocs.io/en/latest/api.html#sftpclient>). Here’s an example client:
>
> import asyncio, asyncssh, sys
>          
> async def run_client():
>      async with asyncssh.connect('localhost') as conn:
>          async with conn.start_sftp_client() as sftp:
>              await sftp.get('example.txt')
>      
> try:
>      asyncio.get_event_loop().run_until_complete(run_client())
> except (OSError, asyncssh.Error) as exc:
>      sys.exit('SFTP operation failed: ' + str(exc))
>
>
> The block_size and max_requests can be added as optional arguments to the sftp.get() in this example, but parallelism will automatically be used for any file larger than 16 KB, allowing for up to 2 MB of data to be outstanding at once (128 requests of 16 KB each), without any threading and all over a single SFTP channel.
>
> This parallel I/O support in AsyncSSH also extends to clients making large read() or write() calls on a remote file. If you call read() or write() with a length of more than the block_size set when a file is opened, these reads or write will automatically be broken into multiple parallel requests. This behavior can be disabled by setting block_size to 0 if you really need to control the exact requests sent on the channel, but by default callers will get the benefit of the extra speed.