[PATCH] Improving sftp client performance

[Dan Kaminsky]

> The openssh sftp client will always upload file blocks in sequence.

Good stuff :-)

> >     Also, I think it was mentioned that the overlapping techniques are
only
> > being used for blocks within one file.  It's *very* common to need to
copy
> > vast numbers of small files from one location to another, and finding
> > efficient means of doing so can sometimes be challenging.  It's
something to
> > keep in mind.
>
> Correct, but I doubt the openssh sftp client is, or ever will be, the tool
> for that.

Shockingly enough, heavily multithreaded FTP has become *the* preferred
method for large scale minifile dumping around here.  LeechFTP uses a
variable number of threads, from five to fifteen to service a unified queue
of files that scales to the thousands.  It ends up being even faster than my
normally preferred method, which is tar-over-ssh.

Now, I bring this up because the one Windows SFTP client I've seen operates
in *exactly* this manner, and is blisteringly fast.  Clearly there's a use
for extending your blocking code across multiple files, but I do admit it
may not be easily done.

Just some perspective.

--Dan