SCP with Resume Feature
rapier at psc.edu
Sun Apr 4 02:14:18 AEST 2021
Parallelizing the transfer isn't a bad idea assuming the goal is to
saturate the path. Its not super hard to do if you are doing it a per
file basis but optimizing the queue can be hard to do right. That said,
using fpart as a feeder to the queues would work. It's how parsyncfp works.
Anyway, the HPN-SSH works hard to saturate the pipe already by using
optimized buffers, parallel ciphers (which need more work - currently
trying to get AES-NI parallelized for CTR and chacha20), disabling the
cipher (post authentication) for non-sensitive transfers, and so forth.
I can push data at around 4Gb/s with a single stream between
On 4/3/21 11:30 AM, hvjunk wrote:
>> On 01 Apr 2021, at 19:50 , rapier <rapier at psc.edu> wrote:
>> The code is available at https://github.com/rapier1/openssh-portable in the resume-scp branch. The patch file is ~1k lines long so I didn't think posting it here would be the right move.
> Given the date this was released, I had to double check… I’m just surprised it didn’t include a “—-parallel=<number>” to speedup the transfers over latency links… much more needed for me transferring data continuously between the northern and southern hemispheres ;(
> openssh-unix-dev mailing list
> openssh-unix-dev at mindrot.org
More information about the openssh-unix-dev