Support for transferring sparse files via scp/sftp correctly?

[Ron Frederick]

On Mar 6, 2025, at 9:38 PM, Damien Miller <djm at mindrot.org> wrote:
> If you want this to happen, I recommend starting by figuring out what
> protocol extensions need to be made, and how to support sparse files
> on system without SEEK_DATA/HOLE - it should be pretty to do this on
> upload without these flags and without extensions.


I was inspired by this thread to add sparse file support to AsyncSSH, on OSes that support SEEK_DATA and SEEK_HOLE. It looks like I should also be able to get this to work on Windows with FSCTL_QUERY_ALLOCATED_RANGES and FSCTL_SET_SPARSE, but I haven’t gotten to that yet.

As Darren Tucker said, the put() operation here can be made to work with any SFTP server. However, an SFTP extension is required to support this for get() or copy(), or the  case where the copy-data extension is used to copy data between files on a remote server without reading and writing it back over the wire.

I’ve defined an extension called "ranges at asyncssh.com <mailto:ranges at asyncssh.com>” which is modeled somewhat after FXP_READDIR for getting valid data ranges in a remote file. Each call can return multiple ranges, but on files with a large number of ranges you may need call this new method multiple times to get the complete list. This allows for the copying to be interleaved with the range requests.

The extension looks like the following:

    uint32          id


-- 
Ron Frederick
ronf at timeheart.net