HiddenStore option may be useful

[Jefferson Ogata]

On 2007-04-15 04:18, William Ahern wrote:
> On Sat, Apr 14, 2007 at 10:54:43PM -0400, Jason wrote:
>> Thomas Blank wrote:
>>> I'm missing a HiddenStore option in OpenSSH, known from some ftp-server 
>>> implementations like ProFTPd.
>>>
>>> Consider the following scenario:
>>> - A process PROCA is frequently polling the directory for a file called 
>>> myfile.txt
>>> - Someone transfers this file via sftp or scp to the directory
>>> - While transfer is going on and the file is not completely written, 
>>> PROCA reads in the file and removes is
>>> -> Corrupt data is seen by PROCA
>>>
>>> Knowing this problem you have to solutions:
>>> 1. PROCA must check if myfile.txt is changing (filesize, mtime...) and 
>>> wait until it does not change any more
>>> 2. sftp and scp use a HiddenStore by writing the file with a unique 
>>> filename (eg. .myfile.txt) and renaming it at the end of the transfer 
>>> (mv .myfile.txt myfile.txt)
>>>
>>> What do you think about this?
>> Why not have PROCA use inotify?
>>
>> See /usr/src/linux/Documentation/filesystems/inotify.txt
> 
> How does that address the race condition? inotify is just a better poll in
> this case. It doesn't even tell you how many processes have an open
> descriptor.
[snip]
> The only real answer, in this case, for race free, provably correct
> behavior, is to hack up sftp. OTOH, if "good enough" is sufficient, you're
> probably already there.

Most systems have fuser. PROCA could either shell out to fuser or do
whatever fuser does on your target platform. This doesn't protect you if
the sftp/scp gets killed, but it comes closer.

An even better option is to do the transfer using tunneled rsync, which
does a HiddenStore-type thing on its own.

Not that I'm against the HiddenStore option. But I don't see why it's
really necessary for this particular case.

-- 
Jefferson Ogata <Jefferson.Ogata at noaa.gov>
NOAA Computer Incident Response Team (N-CIRT) <ncirt at noaa.gov>
"Never try to retrieve anything from a bear."--National Park Service