SCP with Resume Feature

[Jim Knoble]

-- 
jmk

> On Apr 4, 2021, at 11:59, rapier <rapier at psc.edu> wrote:
> 
> [...] I am trying to figure out how to reduce the number of hash operations. Let me lay it out to see if anyone has ideas (aside from using rsync - which I fully support).

Have you considered implementing this using librsync, which is LGPL-2.1?
https://github.com/librsync/librsync

This is the delta library used by rdiff-backup (which you might consider looking at for inspiration (it's Python, but is GPL v2, so use care)).

> [...] I think rsync only computes hashes if the modification time, files sizes, and other file stat data is different. I thought about doing that but since you can rename the target with scp that won't work.

(This behavior can be changed by using the `--checksum` option with rsync, which hashes a file on both ends to determine sameness).

> [...] I also cannot figure out why I can append directly to the target file. After opening the file I'd seek to the end but the bytes would still start at the 0th byte. I'm probably missing something in atomicio. Writing the temp file and then appending works and it's not taking up a lot of cycles but it doesn't feel like the 'right' way to do it.

This certainly does sound suboptimal. Maybe if you post the code?

Cheers.