question on scalability

[Andrey Ermolinskiy]

> This sounds like a TCP problem, not a ssh problem. If the ACK is dropped
> by the server end, then the client should just resend?

The behavior that we observed is that the client end does not automatically
resend the ACK. Instead, the server end goes into a backoff-retry mode, in
which it waits for some time, and then resends the SYNACK to the client.
This causes the client to resubmit its ACK, after which the server makes
anothr attempt to place the connection on the accept queue. The waiting
timeout is intially 1 second and is doubled after each retry.

If, after 5 retries (as defined by tcp_synack_retries tunable), the queue
is still full, the server will quietly give up and will leave the
connection
in SYN_RCVD state forever. This hangs the client command because the client
end has already entered the ESTABLISHED state, and is sitting in a socket
call, presumably waiting for data.

One could argue that it's a problem in the implementation of Linux TCP, and
that the server end should terminate such connections with the RESET flag.
Perhaps this condition hasn't been noticed because most socket application
nowadays use a much larger value for backlog (typically, SOMAXCONN).

The bottom line is that if it is safe to increase the backlog in openssh,
doing this would probably prevent such conditions from occurring.

Regards,

Andrey