File Offsets for SCP (patch revised yet again)

[rapier]

We actually built a test platform (a live ubuntu cd they run on their 
local machine) to automatically run a series of tests  (iperf, npad, 
ndt, etc) across a set of six or seven congestions control algorithms. 
This allows us to quickly determine the best algorithm for the 
particular users network path (along with diagnosing other performance 
issues). The problem is that we can't get all users to boot this ISO 
and, even with and aggressive algorithm we can't always spend as much 
time out of slow start as we want. I'm working on the math now to 
determine the optimal number of concurrent streams for various 
algorithms, bandwidth (1Gb, 10, 40, and 100), and so forth.

At some point we end up running into CPU limits even when we switch the 
cipher to NONE. Of course, if we ever thread the HMAC routine maybe that 
will help. At that point we'd probably run into disk I/O issues... 
Anyway, we're actually planning on using clusters of machines to 
transfer data. One section of the file will be on machine A, one will be 
on B, and so forth.


Philipp Marek wrote:
> On Tuesday 07 December 2010, rapier wrote:
>> The second is that the messaging window size is a limitation for the
>> users I'm supporting (major research institutions usually connected via
>> multiple gig or 10Ge connections). Anything that ends up impeding
>> maximum performance is going to be problematic (that is why I developed
>> the HPN-SSH patch). At some point it may make sense to dynamically grow
>> the message window size (because expecting users to set it correctly is
>> just not going to work (thats why we did the work on the auto-tuning
>> receive buffers)). I may get the funding to do that but I just don't
>> know at this point. So right now I'm going with the quick and dirty fix
>> :)
>>
>> Basically, network performance is where my focus is rather than SSH in
>> and of itself.
> ...
>>>> Again, I don't know if this will be useful to anyone else. We're going
>>>> to be using this to do parallelized transfers of large (250GB+) files
>>>> (using the HPN patch set as well). We're basically trying to avoid
>>>> the slow start penalties on very fast throughputs by splitting the
>>>> transfer up.
> Reading all this I just don't know whether this is just the wrong 
> solution... If you want faster startup, how about using a different TCP 
> scheduling algorithm?
> 
> 	http://www-iepm.slac.stanford.edu/bw/tcp-eval/
> 
> 
> Having a config option to choose the scheduler might be useful to more 
> people, I think - especially as it would do faster transfers automatically, 
> without messing around with partial transmissions.
> BTW that would lower fragmentation on the destination, too.
> 
> 
> Regards,
> 
> Phil