intermittent ssh connect error

[Paul Archer]

We've got set of servers that use AFD (http://www.dwd.de/AFD) to scp files 
to a set of receiving servers. There are about 25 sending machines, and 12 
receiving machines. All the servers are RHEL 4 and 5, with the senders 
running openssh 4.3p2, and the receivers running 3.9p1.

Most files are sent without a problem, but on a pretty regular basis, we 
get "ssh_exchange_identification: Connection closed by remote host" in our 
log files on the senders.
The odd/interesting thing is that while that error often shows up on a 
connection to one machine, we also get bursts, where the error happens on 
connections to several different machines at once. That leads us to think 
it's an issue on the sending side, rather than the receiving side.

I've tried tuning various network paramaters, both on the senders and the 
receivers, but no joy. I wasn't really sure that would help anyway, as 
netstat only reports between 20 and 30 connections (established & 
time_wait) at any given time--although that is almost exclusively ssh.

Any recommendations on where to go from here? What to look for, what 
diagnostics to try?

Paul


--------------------
     Paul Archer
Senior Unix SysAdmin
     Masergy, Inc.
     214-442-8827
parcher at masergy.com
--------------------


PS My apologies if this isn't the appropriate list for this.

PPS This is a sample of the errors we're getting in our AFD log files:
18 21:09:19 <I> hoard10 [0]: 4796416 Bytes send in 8 file(s).
18 21:09:19 <I> hoard09 [2]: 2170880 Bytes send in 26 file(s).
18 21:09:19 <I> hoard04 [2]: 2170880 Bytes send in 26 file(s).
18 21:09:36 <E> hoard03 [0]: Protocol error. ssh is complaining, see next 
message. #28b9df9d (ssh_common.c 427)
18 21:09:36 <E> hoard03 [0]: ssh_exchange_identification: Connection 
closed by remote host
18 21:09:36 <E> hoard03 [0]: SCP connection to 64.47.61.180 at port 22 
failed (-1). #28b9df9d (sf_scp.c 298)
18 21:09:36 <E> hoard01 [0]: Protocol error. ssh is complaining, see next 
message. #ba53f3db (ssh_common.c 427)
18 21:09:36 <E> hoard01 [0]: ssh_exchange_identification: Connection 
closed by remote host
18 21:09:36 <E> hoard01 [0]: SCP connection to 64.47.61.178 at port 22 
failed (-1). #ba53f3db (sf_scp.c 298)
18 21:09:36 <E> hoard11 [0]: Protocol error. ssh is complaining, see next 
message. #1a3b49e6 (ssh_common.c 427)
18 21:09:36 <E> hoard11 [0]: ssh_exchange_identification: Connection 
closed by remote host
18 21:09:36 <E> hoard11 [0]: SCP connection to 64.47.61.132 at port 22 
failed (-1). #1a3b49e6 (sf_scp.c 298)
18 21:09:36 <E> hoard04 [2]: Protocol error. ssh is complaining, see next 
message. #d83bb35 (ssh_common.c 427)
18 21:09:36 <E> hoard04 [2]: ssh_exchange_identification: Connection 
closed by remote host
18 21:09:36 <E> hoard04 [2]: SCP connection to 64.47.61.157 at port 22 
failed (-1). #d83bb35 (sf_scp.c 298)
18 21:09:36 <E> hoard05 [0]: Protocol error. ssh is complaining, see next 
message. #44f6ad16 (ssh_common.c 427)
18 21:09:36 <E> hoard05 [0]: ssh_exchange_identification: Connection 
closed by remote host
18 21:09:36 <E> hoard05 [0]: SCP connection to 64.47.61.156 at port 22 
failed (-1). #44f6ad16 (sf_scp.c 298)
18 21:09:38 <I> hoard12 [0]: 16384 Bytes send in 4 file(s).
18 21:09:38 <I> hoard06 [0]: 16384 Bytes send in 4 file(s).
18 21:09:38 <I> hoard07 [0]: 16384 Bytes send in 4 file(s).
18 21:09:39 <I> hoard08 [0]: 16384 Bytes send in 4 file(s).
18 21:09:39 <I> hoard02 [0]: 16384 Bytes send in 4 file(s).
18 21:09:47 <I> hoard09 [1]: 4812800 Bytes send in 100 file(s).