HPN Patch for OpenSSH 4.2p1 Available

[Chris Rapier]

The performance hit is, initially somewhat suprising, but on reflection 
I think this is mostly going to be dependent on how the systems TCP 
stacks are tuned. I can see performance decreasing in the LAN if the 
receive buffer is too high. An easy test would be to run some iPerfs and 
see if the system TCP receive buffer shows a similar performance hit 
versus setting the the iPerf window to 64k. I'll run some similar tests 
to see if I'm right.

However, this should only be an issue with non-autotuning kernels. 
Autotuning kernels such as linux 2.4.26+. 2.6+, and Windows Vista (nee 
Longhorn) will adjust the receive buffer to maximize throughput. Since 
HPN-SSH is auto-tuning aware this shouldn't be a problem on those 
systems. On non-autotuning kernels appropriate use of the -w option 
should resolve this.

Again, I'll need to test this but I'm pretty sure thats the issue.

Darren Tucker wrote:
> Chris Rapier wrote:
> 
>> As a note, we now have HPN patch for OpenSSH 4.2 at 
>> http://www.psc.edu/networking/projects/hpn-ssh/
>>
>> Its still part of the last set of patches (HPN11) so there aren't any 
>> additional changes in the code. It patches, configures, compiles, and 
>> passes make tests without a problem. I've not done extensive testing 
>> for this version of openssh but I don't foresee any problems.
>>
>> I did run a couple tests between two patched 4.2 installs (one in 
>> switzerland, the other in pennsylvania, usa) and hit around 12MB/s 
>> with the hpn patch and 500KB/s with the standard install. So it still 
>> seems to work as expected.
> 
> 
> Have you done any analysis of the impact of the patch on low-latency/BDP 
> links (ie LANs)?
> 
> I've been doing some work with parts of the HPN patch.  For scp'ing to 
> and from localhost and LANs (see below), the scp changes on their own 
> (actually, a variant thereof) shows a modest improvement of 2-3% in 
> throughput.  That's good.
> 
> For the entire patch (hpn11), however, it shows a significant *decrease* 
> in throughput for the same tests: 10% slower on OpenBSD to localhost, 
> 12% slower on Linux to localhost and 18% slower Linux to OpenBSD via a 
> 100Mb/s LAN).  That's bad.  I would imagine LANs are more common than 
> high BDP networks that your patch targets :-)

I'll check this of course. We don't have any OpenBSD systems though but 
I'll try to find a spare box we can blow it on to.

> I suspect that the buffer size increases you do are counterproductive 
> for such networks.  I also suspect that you could get the same 
> performance improvements on high-BDP links as you currently do by simply 
> increasing the channel advertisements without the SSH buffer changes and 
> relying on the TCP socket buffer for output and decrypting quickly for 
> input but I'm not able to test that.

Of course is possible to increase performance through other means. 
Transferring data in parallel for example, is a great way to do this. In 
fact the suggestions you made are ones we were planning on implementing 
in addition to the buffer hack. Especially now that we really are CPU 
limited as opposed to being buffer limited.

However, that seems, in my view at least, to be an overly complicated 
way of going about it. The buffer hack is pretty straightforward and 
well known - the concept is laid out in Stevens TCP/IP Illustrated 
Volume 1 after all.

> Test method, with a 64MB file:
> $ time for i in 1 2 3 4 5 6 7 8 9 0; do scp -ocompression=no -o 
> ciphers=arcfour /tmp/tmp localhost:/dev/null; done

I'll try this out and let you know what I find. Could you let me know 
what you had your tcp receive buffer set to when you tried these tests? 
Optimally for these local tests it should be set to 64k.

By the way - I just ran the above test and didn't see any sort of 
substantive difference. There was a slight edge to the HPN platform 
*but* that was probably due to the scp hack.

Of course, in this case the problem is likely to be disk speed limits 
(I'm on a powerbook at the moment and its disks are slow). Bypassing scp 
and doing a direct memory to memory copy is probably the right methodology.