Support for transferring sparse files via scp/sftp correctly?

[Cedric Blancher]

On Tue, 4 Mar 2025 at 21:22, Chris Rapier <rapier at psc.edu> wrote:
>
>
>
> On 3/4/25 05:34, Philipp Marek via openssh-unix-dev wrote:
> >> Does OpenSSH scp/sftp mode transfer sparse files correctly, i.e. are
> >> holes skipped and not transferred as chunks of 0 bytes? [1]
> >>
> >> We're asking about sparse files in the >= 1PB range, which consists of
> >> multi-TB holes with around 600-2000GB of valid data.
> >
> >
> > Perhaps rsync would be a good fit here,
> > it supports --sparse.
> > _______________________________________________
> > openssh-unix-dev mailing list
> > openssh-unix-dev at mindrot.org
> > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>
> I think one of the issues you are going to face is that SEEK_DATA and
> SEEK_HOLE don't seem to be currently supported under OpenBSD. Since
> that's the home OS for OpenSSH this could create portability issues.
> While you can get around that with the judicious use of defines it means
> that the feature set will start to shift between different OSes.

OpenBSD unfortunately does not implement so many other APIs. But other
OS do implement SEEK_DATA+SEEK_HOLE, including FreeBSD, Linux,
Solaris, Illumos and even Cygwin. Even NFS has a SEEK to lookup holes
and data sections in files.
SEEK_HOLE+SEEK_DATA are also now part of the POSIX standard, so IMO it
is time to face the bug that sparse files are not handled correctly
and fix it

>
> Personally I think it's a good idea and I may explore it for HPN-SSH but
> I think it's going to be a hard sell for the OpenBSD community.

Selling points:
- Users need it
- Sparse files were always part of UNIX
- It's part of the POSIX standard:
https://pubs.opengroup.org/onlinepubs/9799919799/functions/lseek.html

Ced
-- 
Cedric Blancher <cedric.blancher at gmail.com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur