ssh / scp slow on 10GBE

[Tomas Kuthan]

On 01/04/13 12:44, Stefan Priebe wrote:
> Hi,
> 
> Am 03.01.2013 20:19, schrieb Iain Morgan:
>> I think you have a typo above. You presumably mean 600-700 MB/s in the
>> case of OpenSSL using AES-NI, but do you mean MB/s or Mb/s in the scp
>> case?
> uups i mean MB in both cases.
> 
>> How are you measuring the ssh/scp performance? Is this over a LAN or WAN
>> connection?
> LAN
> 
>> On the other hand, if you are getting 76 MB/s, then you may have some
>> other bottlenecck to address. You might try doing a dd through ssh to
>> factor out scp and disk I/O. For example,
>>
>> % dd if=/dev/zero bs=1048576 count=4096 | ssh somehost dd of=/dev/null
> #~ dd if=/dev/zero bs=1048576 count=4096 | ssh -c aes128-cbc 10.255.0.24
> dd of=/dev/null
> 4096+0 records in
> 4096+0 records out
> 4294967296 bytes (4,3 GB) copied, 19,3355 s, 222 MB/s
> 8388608+0 records in
> 8388608+0 records out
> 4294967296 bytes (4,3 GB) copied, 18,6107 s, 231 MB/s
> 
> But:
> # iperf -c 10.255.0.24 -t 30 -d
> ------------------------------------------------------------
> Server listening on TCP port 5001
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
> ------------------------------------------------------------
> Client connecting to 10.255.0.24, TCP port 5001
> TCP window size:   990 KByte (default)
> ------------------------------------------------------------
> [  4] local 10.255.0.22 port 44006 connected with 10.255.0.24 port 5001
> [  5] local 10.255.0.22 port 5001 connected with 10.255.0.24 port 37393
> [ ID] Interval       Transfer     Bandwidth
> [  4]  0.0-30.0 sec  34.6 GBytes  9.90 Gbits/sec
> [  5]  0.0-30.0 sec  33.9 GBytes  9.70 Gbits/sec
> 
> and
> openssl aes-128-cbc speed:
> 
> The 'numbers' are in 1000s of bytes per second processed.
> type             16 bytes     64 bytes    256 bytes   1024 bytes   8192
> bytes
> aes-128-cbc     648664.33k   688924.90k   695855.45k   700784.64k
> 704027.06k
> 
> Greets,
> Stefan
> 
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev at mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Hi Stefan,

in order for ssh receive window to not impose an additional bottleneck,
it should be set to be roughly equal to bandwidth-delay-product. Taking
into account your measured bandwidth of about 1.2 GBps, and assuming
round-trip-time on LAN lesser or equal to 1 ms, OpenSSH's hard-coded
window size 2 MB should be just about enough:
1.2 GBps * RTT <= 1200 MBps * 0.001 s < 2 MB

Just to make sure, could you please measure round-trip-time on the
connection? Simply running ping will do...
If the result is not 2 ms or higher, ssh window side is most probably
not the culprit.

Part of the gap between encryption speed and the dd measurement can be
accounted to checksumming, some to initial key-exchange and rekeying.
The gap still feels quite wide though.

I don't know, how your initial measurements (76 MBps) were done. But if
it involved copying multiple smaller files, this could very well be scp
protocol overhead. In that case tar could help.

Cheers,
Tomas