IPQoS EF packets blocked by some provider(s) / Login to OpenSSH >= 10.1 servers stuck in preauth

[Oliver Freyermuth]

Dear fellow OpenSSH friends,

this is not so much a bug report, but a troubleshooting report in the hope it might be helpful to others using OpenSSH in a similar environment.


TL;DR: Some providers seem to block incoming packets with the QoS marking "Expedited Forwarding", effectively blocking login to OpenSSH servers >= 10.1 with default configuration.
        "IPQoS none" in sshd_config (server-side) works around this.


Full story:
I recently upgraded my first server to OpenSSH 10.1_p1 (and subsequently 10.2_p1) and found that starting with 10.1_p1 I was unable to log in from my home network (neither via IPv4 nor IPv6), independent of the client version. From other network locations, it worked fine, even carrying a mobile client from home into another network.

Checking debug logs, my client at home got stuck right before receiving the banner from the server, and the server session got stuck in pre-auth.
tcpdumping / Wiresharking revealed that the packet with the banner and the first KEX packet sent by the server did not arrive with the client. Neither the original packets nor subsequent retransmissions arrived, leading to a "connection stuck in pre-auth".

As I did not have an immediate idea on which change caused this (network packets looked "similar enough" between 10.0 and later), I did a bisect and found that the new IPQoS functionality ( https://github.com/openssh/openssh-portable/commit/289239046b2c4b0076c14394ae9703a879e78706 ) triggered this.
And indeed, I could confirm that the packets which never arrived at my client had Expedited Forwarding / EF set, i.e. DSCP 46.
I could have grasped that new feature from the release notes, but sometimes you have to learn the hard way...

Indeed, setting:
  IPQoS none
in sshd_config on the server and restarting it fixes things for me. I can again log in from home to this (self-managed) SSH server, but of course I will remain locked out from other > 10.0 servers in my current setup.

And indeed, running:
  ping -Q 184
(this is 46 << 2) from "outside" against the WAN IP of my CPE / home router (directly connected to a fibre) yields full packet loss.
tcpdumping on the external interface of the router shows that the packets do not arrive there, and I tried from various locations (which can ping each other that way), so I deduce my provider is silently dropping these packets (in my case, I have CG-NAT for IPv4 and native IPv6, they drop this for both, while of course for IPv4 I can only test via a tracked connection through their NAT).

Since my provider serves >1.5 million customers here in Germany, I presume this might hit others out there (maybe even with other providers). Sadly, getting through their first level support is almost impossible, so I have only a small hope of them fixing it.
In case it hits others, I do hope this troubleshooting here may prove useful to get a temporary workaround or maybe help others to escalate to their provider in case this is possible for them.


I'm not sure OpenSSH can improve on this situation. Reading through RFC 3246 section 3 [0], maybe it should, as it states:
  "If two adjacent domains have not negotiated an EF rate, the downstream domain SHOULD use 0 as the rate (i.e., drop all EF marked packets)."
This could indicate that this might happen in other cases, which then essentially blocks interactive SSH in default server configurations.


Cheers and hope this helps,
	Oliver

[0] https://www.rfc-editor.org/rfc/rfc3246#section-3