About rsync over SSH and compression

[Brent L. Bates]

     I've been doing a lot of testing of different options for both rsync
and ssh.  For our environment, turning compression off in rsync and on in
ssh has given us the best results.  Turning compression on in ssh gave us a
small improvement compared to the large slowdown that compression gave us
in rsync.  For us, we also made sure that the checksum option in rsync was
off.  It caused a major slowdown for us.  It was doing a checksum on every
file before deciding if needed to transfer the file.  That was killing us.
We are transferring a wide variety of sizeed files, from small KB sized to
GB sized files.  We are also transferring 10's of thousands of files
between states.  I hope this helps some.


Brent L. Bates


On Tue, Jun 25, 2019, 12:06 PM JCA <1.41421 at gmail.com> wrote:

> Rsync supports the capability of compressing data before sending. So does
> OpenSSH. It would be probably be a waste of resources and time to enable
> both compression capabilities at the same time, but it is not clear to me
> whether, in general, it makes better sense to enable rsync compression or
> SSH compression.
>
> My first thought would be that SSH compression might yield better results,
> on the ground that SSH will try to cram as much data as possible in a
> channel data message, within the limitations imposed by the maximum SSH
> message size and the current window size. On the other hand, rsync might
> (and 'might' is the keyword here) resort to individual
> SSH2_MSG_CHANNEL_DATA messages for particular deltas, thus giving the
> compression code smaller amounts of data to play with every time.
> Additionally, SSH compression will be able to compress the rsync protocol
> control data, which rsync will not be able to compress (right?)
>
> Feedback on the issues above will be much appreciated.
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev at mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>