HPN SSH

[Chris Rapier]

Avnish & OpenSSH Community,

I'm sorry this has taken so long to reply to. I've been out of the 
office for the past 3 week dealing with nuptials and such.

Avnish Bhatnagar wrote:
> Hello,
> 
> I know this has come up before; but is the HPN patch (or elements thereof)
> currently being considered for integration in to the OpenSSH code base? 

I don't know. I do know that the scope of my patch is pretty broad and 
getting it integrated at this point would require a certain amount of 
effort to have it properly vetted. I understand how little time people 
have to deal with these sort of things and how different people and 
users have different priorities. However, even if the OpenSSH dev team 
doesn't like the exact code I and Mike Stevens developed I do believe 
that the concept is worth pursuing and I'm willing to do what I can to 
provide necessary test beds if anyone on the dev team is interested.

Personally, I don't have anything emotionally invested in the code 
itself. I just believe that incorporating the concept would be useful to 
a large number of users - both in the higher end environments and in the 
growing number of home environment with 10Mb/s+ connections.


> Are
> there pending issues (buffer management, none cipher, etc) which still need
> to be addressed?

I believe most of these issues have been addressed but I might be 
mistaken. The none cipher usage maybe a show stopper but I have always 
seen that as a secondary feature to the performance enhancements.

> We have been using HPN-SSH for over a year now, and like others, have
> observed significant performance improvement over standard OpenSSH.  I can
> scp a 1 GB test file between two HPN-SSH LAN hosts at 700 Mbps (<1 ms
> latency).  And over a cross-country high-BDP WAN link, I'm able to achieve
> over 500 Mbps (85 ms latency).  These single-stream scp transfers were run
> on well-tuned Linux kernels 2.6.15 (or higher) with the arcfour cipher.
> (I'll be happy to provide more details about these tests upon request.)  I'm
> not sure how 'typical' my results are, but they represent an order of
> magnitude improvement over stock OpenSSH.  While the improvement tends to
> vary among different platforms, I have never observed a performance
> degradation.

Actually those increases are pretty typical. Its not perfect and there 
are still some aspects that can prevent a user from seeing the rates 
they might see with gridFTP or some other dedicated bulk data mover but 
in terms of simplicity, functionality, and security SSH, and the OpenSSH 
implementation, is tough to beat. We've actually been thinking about 
extending our usage of it to make it more of a middleware transport 
mechanism for some applications.

> We recommend HPN SSH to our users who need to (securely) transfer their bulk
> scientific datasets ranging in size of hundreds megabytes to one terabyte;
> so naturally, performance is very important for them.  But they (or their
> sysadmins) are often reluctant to deploy software which represents a
> deviation from a standard distribution...and the maintenance issues that
> follow.

Which is understandable and one of the questions I frequently field on 
this. Is it compatible and maintainable? Is it secure? Will it be part 
of the main code base? (yes, yes, and I don't know). Still, there are a 
growing number of people that have started using it as their default 
distribution. The Data Working Group of the TeraGrid recently made it a 
mandatory component for data transport servers. So adoption is growing, 
at least in the HPN community.

Chris Rapier
Pittsburgh Supercomputing Center