[PATCH] Improving sftp client performance

[mouring at etoh.eviladmin.org]

On Sat, 5 Jan 2002, Dan Kaminsky wrote:

> > The openssh sftp client will always upload file blocks in sequence.
>
> Good stuff :-)
>
> > >     Also, I think it was mentioned that the overlapping techniques are
> only
> > > being used for blocks within one file.  It's *very* common to need to
> copy
> > > vast numbers of small files from one location to another, and finding
> > > efficient means of doing so can sometimes be challenging.  It's
> something to
> > > keep in mind.
> >
> > Correct, but I doubt the openssh sftp client is, or ever will be, the tool
> > for that.
>
> Shockingly enough, heavily multithreaded FTP has become *the* preferred
> method for large scale minifile dumping around here.  LeechFTP uses a
> variable number of threads, from five to fifteen to service a unified queue
> of files that scales to the thousands.  It ends up being even faster than my
> normally preferred method, which is tar-over-ssh.
>
Doing multiple ftp streams is what you are refering to.  That could be
done, but it does not improve a single file transfer.  I think I would
perfer to get this working first and then look at if we want to improve
the "mget *" or "mput *" case.

Because that code would be at a higher level in some respects.

- Ben