[PATCH 0/5] ZSTD compression support for OpenSSH

[Sebastian Andrzej Siewior]

On 2020-09-07 11:21:13 [+1000], Darren Tucker wrote:
> The zstd part would be a larger discussion because we would need to
> either carry it as a Portable patch or have zstd added to OpenBSD
> base, and I don't know if that would be accepted.  Do you have any
> performance numbers for zstd in this application?

A key stroke is here 10 bytes of raw data which zstd compresses usually
into 10 bytes while zlib manages to squeeze it into 5 bytes. This leads
to better compression ratio for zlib in ssh's accounting (visible in
verbose mode after connection terminates). The data length, that will be
transferred over the wire, is the same for 5 and 10 bytes data after the
crypto part (with padding and so on). 

Regarding statistics, do you have anything specific in mind?
A ~100MiB file copied with scp over a 10MBit link (the percentage number
after CPU indicates the CPU load as observed by top):
(scp file server:)
zstd
| CPU, ssh 4-6%, sshd 7-16%
| file  100%  107MB  11.5MB/s   00:09
| Transferred: sent 11144324, received 30760 bytes, in 9.5 seconds
| Bytes per second: sent 1176963.9, received 3248.6
| debug1: compress outgoing: raw data 112653640, compressed 11056148, factor 0.10
| debug1: compress incoming: raw data 9281, compressed 12113, factor 1.31

zlib
| CPU, ssh 30%, sshd 7-16%
| file  100%  107MB  14.0MB/s   00:07
| Transferred: sent 9037812, received 21984 bytes, in 7.8 seconds
| Bytes per second: sent 1159233.8, received 2819.8
| debug1: compress outgoing: raw data 112653676, compressed 8949791, factor 0.08
| debug1: compress incoming: raw data 8885, compressed 4382, factor 0.49

A 530MiB file over a ~200MBit link:
(scp server:file file)

zlib
| CPU, sshd 100%, ssh 30%
| t 100%  537MB  59.3MB/s   00:09
| Transferred: sent 100148, received 45178992 bytes, in 9.2 seconds
| Bytes per second: sent 10924.6, received 4928350.1
| debug1: compress outgoing: raw data 44367, compressed 21128, factor 0.48
| debug1: compress incoming: raw data 563267178, compressed 44746500, factor 0.08

zstd
| CPU, sshd 30-40%, ssh 26-28%
| t 100%  537MB  82.6MB/s   00:06
| Transferred: sent 144496, received 55714260 bytes, in 6.6 seconds
| Bytes per second: sent 21789.3, received 8401454.6
| debug1: compress outgoing: raw data 46014, compressed 61226, factor 1.33
| debug1: compress incoming: raw data 563267187, compressed 55281740, factor 0.10

incompressible data
zlib
| CPU, sshd 70%, ssh 14%
| u 100%  300MB  22.5MB/s   00:13
| Transferred: sent 57068, received 315112228 bytes, in 13.5 seconds
| Bytes per second: sent 4236.6, received 23393315.6
| debug1: compress outgoing: raw data 24981, compressed 11877, factor 0.48
| debug1: compress incoming: raw data 314745446, compressed 314860750, factor 1.00
 
zstd
| CPU, sshd 18%, ssh 12%
| u  100%  300MB  21.6MB/s   00:13
| Transferred: sent 79060, received 315111884 bytes, in 14.0 seconds
| Bytes per second: sent 5644.4, received 22496976.5
| debug1: compress outgoing: raw data 24981, compressed 33182, factor 1.33
| debug1: compress incoming: raw data 314745500, compressed 314802781, factor 1.00

Sebastian