Possible feature request: sync_on_close mode for scp/sftp uploads

[Jeremy Friesner]

Hi Bryan,

Many thanks, that works like a charm for sftp sessions... I can't seem to get it to work for scp uploads though.  Is there some nuance that I'm missing?

For reference, here is what I put at the end of my /etc/ssh/sshd_config file:

Subsystem scp /usr/lib/openssh/scp-and-sync
Subsystem sftp /usr/lib/openssh/sftp-server-and-sync

..... and the output below shows the two bash scripts I added and referenced above:

Wild-Tracks-1:~# ls -l /usr/lib/openssh/*sync
-rwxr-xr-x 1 root root 111 Jul  1 01:50 /usr/lib/openssh/scp-and-sync
-rwxr-xr-x 1 root root 131 Jul  1 01:50 /usr/lib/openssh/sftp-server-and-sync

Wild-Tracks-1:~# cat /usr/lib/openssh/sftp-server-and-sync 
#!/bin/sh
echo "BEGIN SFTP SESSION!" >>/tmp/sftp_foo
/usr/lib/openssh/sftp-server $*
echo "END SFTP SESSION!" >>/tmp/sftp_foo
sync

Wild-Tracks-1:~# cat /usr/lib/openssh/scp-and-sync         
#!/bin/sh
echo "BEGIN SCP SESSION!" >>/tmp/scp_foo
/usr/bin/scp $*
echo "END SCP SESSION!" >>/tmp/scp_foo
sync

-Jeremy

On Jun 30, 2010, at 6:11 PM, Bryan Whitehead wrote:

> Try something with "Subsystem" in sshd_config. Where you redefine sftp
> to be something like "sftp-server && sync".
> 
> Might need to embed that in a python/bash/perl script so your
> subsystem isn't a bash expression... ;)
> 
> On Wed, Jun 30, 2010 at 4:06 PM, Jeremy Friesner <jaf at meyersound.com> wrote:
>> Hi all,
>> 
>> First, my apologies if this is not the right forum for this... if there is a more appropriate place to make OpenSSH feature requests, or if this feature already exists, please let me know.
>> 
>> Some background:  My company makes an 'embedded' audio server box that runs Debian Linux, and one of the product's features is that users can upload files to the server via SSH, using their favorite SSH-compatible file-transfer program (e.g. command-line scp/sftp, CyberDuck, WinSCP, PSCP, etc).
>> 
>> That works fine, but one issue we have is that this box needs to be robust in the face of uncontrolled shut-downs:  in particular, since it is a headless "black box", our users expect to be able to just cut power to the box, without logging in and executing a "/sbin/shutdown -h now" first.  If cut power soon after completing an scp/sftp upload, however, they often find that last of their uploaded files end up truncated, because some of the uploaded data was cached in RAM and never got flushed to the disk before the power was cut.
>> 
>> I know one possible solution to this problem would be to mount the filesystems with the "sync" flag, so that all writes to the filesystem would be done synchronously.  However, I'm hesitant to do that because there are other processes running on the box (e.g. real-time audio recording) whose performance might be adversely by forcing every write() call to be synchronous.
>> 
>> Given that, my question is:  is there an option (or can an option be added) to force sshd to call fsync() on its uploaded-file's file-descriptors just before it calls close() on them?  That way when an scp/sftp upload completes, our users could be reasonably certain that all the uploaded data would make it onto the disk in the next second or two.  Perhaps a "sync_on_close" keyword in the /etc/sshd_config file would activate this behavior?  (I don't think it should be active by default, since that could unnecessarily affect performance in use cases where uncontrolled server shutdowns are not an issue)
>> 
>> Thanks for your time,
>> 
>> Jeremy Friesner
>> Meyer Sound Laboratories Inc.
>> 
>> _______________________________________________
>> openssh-unix-dev mailing list
>> openssh-unix-dev at mindrot.org
>> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>>