SCP with Resume Feature

rapier rapier at psc.edu
Fri Apr 2 04:50:22 AEDT 2021


Howdy all,

I know development on SCP is discouraged but being that it's still in 
wide use I thought I would do some work some of my users have been 
asking for and allow SCP to resume from a partial transfer.

When enabled the resume function will compare the source and target 
files. The source host will and target host will pass MD5 hashes back 
and forth in order to compare the hash of the file fragment on the 
target and the file up to the length of the source file. If these hashes 
match the source will seek to the appropriate byte position and start 
sending the rest of the file. This will then be appended to the target.

Example:
An scp transfer of a 128,000 byte file terminates after transferring 
64,000 bytes. The user then issues the SCP command again with the -R 
(resume) option. The hash of the 64k fragment on the target is compared 
to the hash of the source file up to 64k. The hashes match so the source 
seeks to the 64,001th byte and then starts send the remainder of the 
file. This is written to a temp file on the target which then appends it 
to the target after the end of the transfer.

If the hash doesn't match then then entire source file is copied to the 
target and replaces the fragment.

In the event that the source and target files have the same size and 
same hash then the file is skipped (no transfer of data). This end sup 
being handy if the user is doing a recursive transfer or using a wildcard.

Both sides have to be using a resume enabled SCP for this to work. Since 
scp uses the first scp in the users path I've added an option that lets 
users define which scp to use on the remote side. This can be set with 
'-s' e.g. 'scp -R -s /opt/openssh/bin/scp' foo* you at host:~/foofiles/'

The code is available at https://github.com/rapier1/openssh-portable in 
the resume-scp branch. The patch file is ~1k lines long so I didn't 
think posting it here would be the right move.

Note: This is the first functional implementation of this so there are a 
lot of debug statement and I am sure there are ways of improving this. I 
also know that users can just use rsync. That said, this was a 
deliverable and the users wanted this. So I did it. Anyway, I'm open to 
any and all suggestions, modifications, criticism and so forth.

Thanks,

Chris


More information about the openssh-unix-dev mailing list