djm at mindrot.org
Thu Sep 18 08:12:12 EST 2003
On Wed, 2003-09-17 at 19:18, Markus Friedl wrote:
> we could modify the protocol and implement
> rolling checksums like Niels Provos suggests:
> MD5_CTX ctx1, ctx2;
> while new page in data
> MD5_Update(&ctx1, newpage, pagesize)
> ctx2 = ctx1;
> MD5_Final(digest, &ctx2)
> if (compare with remote not equal)
> end while
> continue data transfer.
That is a nice idea. Poor man's rsync :)
> On Wed, Sep 17, 2003 at 11:12:36AM +0800, Dmitry Lohansky wrote:
> > Hello openssh@
> > I thought about sftp's reget/reput commands.
> > Several days ago, Damien Miller write to tech at openbsd.org (it was
> > reply for my letter):
> > > Herein lies a problem which is not easy to detect or solve. For
> > > performance reasons, the sftp client does pipelined reads/writes when
> > > transferring files. The protocol spec allows for a server to process
> > > these requests out of order. For example:
> > > client server
> > > ------ ------
> > > open file your file handle is "blah"
> > > gimme bytes 0-8191
> > > gimme bytes 8192-16383
> > > gimme bytes 16384-24575
> > > gimme bytes 24576-32767 here are bytes 24576-32767
> > > close file here are bytes 16384-24575
> > > here are bytes 8192-16383
> > > here are bytes 0-8191
> > > close successful
> > > If the client writes the bytes our in the order they are received (which
> > > it probably should, to avoid buffering large amounts of data) then an
> > > interruption will leave a full-length, but "holey" file on disk. There
> > > is no general way to determine how to do resume such a transfer.
> > > The best the client can do to make transfers resumable is ftruncate()
> > > the file at the highest contiguous byte received. This will stop the
> > > potential corruption on resume.
> > This is good method, but if client crash, we also may get a "hole".
> > What your think about next way?
> > Storing extra-data at the end of file, for example:
> > <---orig-part-><-extra->
> > [*][ ][*][ ][*][*******]
> > <---------file--------->
> > where [*] - already loaded data, [ ] - not yet
> > In extra part, we may store which block was already loaded and it
> > offset and size. After download, extra part will be removed.
> > Comments?
> > --
> > Dmitry Lohansky
More information about the openssh-unix-dev