ssh / scp slow on 10GBE

[Tomas Kuthan]

Hi Stefan,

sorry for the list-only reply.

Some answers in-line...

On 01/06/13 20:00, Stefan Priebe wrote:
> Hi Tomas,
> 
>>Just to make sure, could you please measure round-trip-time on the
>>connection? Simply running ping will do...
> 
> While running:
> #~: dd if=/dev/zero bs=1048576 count=8192 | ssh -c aes128-cbc -m
> umac-64 at openssh.com 10.255.0.24 dd of=/dev/null
> 8192+0 records in
> 8192+0 records out
> 8589934592 bytes (8,6 GB) copied, 21,8169 s, 394 MB/s
> 16777216+0 records in
> 16777216+0 records out
> 8589934592 bytes (8,6 GB) copied, 21,1476 s, 406 MB/s
> 
> These are the ping stats:
> 22 packets transmitted, 22 received, 0% packet loss, time 20996ms
> rtt min/avg/max/mdev = 0.060/0.103/0.176/0.026 ms

Thanks for the confirmation. In that case window size is sufficient.

So most probably the transfer speed in your case is CPU-bound. You may
want to run 'openssl speed' also on the receiving side, just to make
sure. Alternatively you could compare this result with the same test run
over localhost.

>>If the result is not 2 ms or higher, ssh window side is most probably
>>not the culprit.
> OK thanks for this info.
> 
>>Part of the gap between encryption speed and the dd measurement can be
>>accounted to checksumming, some to initial key-exchange and rekeying.
>>The gap still feels quite wide though.
> Now it's 400MB/s using ssh and 1230MB/s when using a direct TCP connection.

Well, you certainly won't get anything faster than your 700 MB of
openssl aes speed results. You cannot transmit data faster, than you
encrypt them. And even this upper bound is not reachable, because of
checksumming overhead.

In fact, taken into account the relative improvement when switching to
umac-64 at openssh.com (230 -> 400 MBps), it is quite plausible, that all
what remains, is due to checksum computation.

Looks like 400 MBps is all you can get in current situation. Unless off
course, you are willing to condescend to using weaker crypto (arcfour),
which cannot be encouraged (I suppose, integrity and confidentiality are
required).

There is a potential in enhancing OpenSSH to do some tasks in parallel.
MACs and encryption could be done at the same time. Or, for some modes
of encryption, counters can be pre-computed in advance. But in the past
OpenSSH developers expressed strongly against threads...

Best regards,
Tomas

>> don't know, how your initial measurements (76 MBps) were done. But if
>>it involved copying multiple smaller files, this could very well be scp
>>protocol overhead. In that case tar could help.
> Yes sorry scp and a disk was involved. I'm now talking about 400MB/s vs.
> 1230MB/s.
> 
> Greets,
> Stefan