High Performance SSH/SCP - HPN-SSH when?

[Chris Rapier]

Darren Tucker wrote:

>>> I think this is another symptom of the HPN patch letting the buffers get
>>> way too big under some conditions, then ssh spents a disproportionate
>>> amount of time compacting those buffers.
>> Assuming this is true it doesn't address the performance issue with an 
>> unmodified OpenSSH under cygwin.
> 
> That's true, I just didn't want people spending time analyzing (what I
> think is) the wrong thing.

I agree which I why I brought it up :) I'm really just wondering if the 
problem under cygwin is a memcpy issue.

> [...]
>> I will try find out exactly how big the buffer is actually getting in 
>> this. I just need to identify exactly which buffer I need to be paying 
>> attention to :) I'll also just drop MAX_HPN_LEN to 10MB and see what 
>> happens.
> 
> The existing buffer limit is 10MB so unless you have paths with a
> BDP/TCPWINSZ of 10MB or higher (do you?) then I think you'll see your
> throughput go up a little or your CPU and memory utilization go down a
> little.

Great that 10MB? Oh yes, frequently. I know its not common in the really 
real world yet but we're working on 30Gb/s end to end paths at this 
point. There are some specialized transfer tools to make better use of 
these paths but they tend to be a bit of a pain (various globus tools 
and GridFTP for example). Obviously, most people won't be working on 
this but again, its just a matter of time until 10MB is too small. It 
might be another 5 years but gigabit connectivity to the home is coming 
- its already pretty common on college campuses.

However, you are right in that the current size is probably far to high 
though at .5GB. I think our original thinking is that in almost all 
situations the buffer wouldn't grow that high and we might as well 
future proof it as much as possible.

> At the moment, you do this in channel.c:
> -limit = MIN(limit, (BUFFER_MAX_LEN - BUFFER_MAX_CHUNK - CHAN_RBUF));
> +limit = MIN(limit, (BUFFER_MAX_HPN_LEN-BUFFER_MAX_CHUNK-CHAN_RBUF));
> 
> Instead, I suggest leaving the existing line and adding something like:
> 	limit = MIN(limit, gettcpwinsz());
> or
> 	limit = MIN(limit, 2 * gettcpwinsz());

Okay, I'll give that a try.

> That way the buffer utilization would scale with your window size.
> 
> BTW hpn12 doesn't seem to be on your web page?

Its there, just not advertised until I resolve this issue. :)
http://www.psc.edu/networking/projects/hpn-ssh/openssh-4.3p2-hpn12.diff


> If a similar bottleneck applies to pipe/socketpairs, it may also be work
> increasing the IO buffer in scp and seeing if that helps your
> performance (the HPN patch bumps it from 2k to 16k, you may want to bump
> it to 64k).

I know we got a 2-3% reduction in cpu when we made that switch but I 
don't think it actually scaled linearly. Of course, this was under linux 
and os x so the effect under cygwin or *BSD might be much different.

Chris