Rsync of large file seems to cause "message authentication code incorrect"

[Lars Noodén]

I am consistently seeing a "message authentication code incorrect" error 
while running Rsync over OpenSSH.

This script, and similar manual invocations, stopped working a few days 
ago, since Saturday basically after years in production.  The transfer 
seems to stop whenever Rsync gets to one of the larger ( ~ 900 MB 
gzipped ) files.  With increased verbosity for both Rsync and OpenSSH, I 
see the following abort error message consistently then:

...
debug2: channel 0: window 1980256 sent adjust 116896
ssh_dispatch_run_fatal: Connection to xx.yy.zz.aa port 22: message 
authentication code incorrect
rsync: connection unexpectedly closed (850600663 bytes received so far) 
[receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(232) 
[receiver=3.2.7]
rsync: connection unexpectedly closed (12005395 bytes received so far) 
[generator]
rsync error: unexplained error (code 255) at io.c(232) [generator=3.2.7]
rsync: [generator] write error: Broken pipe (32)


That happens each and every time now when running Rsync over SSH using 
the following options:

rsync -arvvAH \
	-e "ssh -v -v -i ${key}" \
	--link-dest=./backup/${d2}/ \
	--acls \
	--sparse \
	--files-from=./backup.files \
	--delete \
	bkup at example.org:/ ./backup/${d1}/

Without the verbosity, Rsync stops with the following error, which is 
basically the same:

...
ssh_dispatch_run_fatal: Connection to xx.yy.zz.zz port 22: message 
authentication code incorrect
rsync: connection unexpectedly closed (1159808568 bytes received so far) 
[receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(232) 
[receiver=3.2.7]
rsync: connection unexpectedly closed (12007275 bytes received so far) 
[generator]
rsync error: unexplained error (code 255) at io.c(232) [generator=3.2.7]
rsync: [generator] write error: Broken pipe (32)


I've verified that the files in the ./backup.files text file all exist 
on the remote system.  Interactive SSH seems not to run into any 
connection problems, so far, and those interactive sessions can last 
(hours at a time) until I close them and don't disconnect on their own. 
I've also tried setting Rsync's --max-alloc to 0 and reducing --bwlimit 
as low as 500 in the script to no avail.  I've also tried setting 
ServerAliveCountMax to 6 and ServerAliveInterval to 30 in ssh_config(5) 
on the local system.  The time it takes until disconnection can range 
from less than a minute to almost four minutes.  However, although the 
time when the break occurs is unpredictable, the break itself always 
happens lately.

What should I be looking at?

/Lars

PS.  I have on the following on the remote system:

$ apt-cache policy rsync | head -n 2
rsync:
   Installed: 3.2.7-1+deb12u2

$ apt-cache policy openssh-server | head -n 2
openssh-server:
   Installed: 1:9.2p1-2+deb12u7

And I have on the following on the local system:

$ apt-cache policy rsync | head -n 2
rsync:
   Installed: 3.2.7-1ubuntu1.2

$ apt-cache policy openssh-client | head -n 2
openssh-client:
   Installed: 1:9.6p1-3ubuntu13.14

And the first hop on the LAN is the following set up.  pf.conf remains 
unchanged for ages.

$ uname -mrsv
OpenBSD 7.8 GENERIC.MP#167 amd64