occasional ssh hang

[mhaverka at kcp.com]

I am using OpenSSH on Redhat 6.2 (Intel) and Solaris 2.6 (Sparc).  I have a
job on the linux machine that ssh's to the Solaris machine every 20 seconds
or so and runs uptime.  The problem is that after many iterations of this,
ssh will occasionally hang, and require a kill -9 to get rid of the
process.  The problem happens with both protocol version 1 and 2, but it
seems to happened more often with version 2.  I have noticed this problem
with both 2.5.2p2 and 2.9p1.

Below is ssh -v output from a successful run, and a run where ssh hangs.
Both are done with protocol version 2.

ssh -2 -x -T -n -v -o 'BatchMode yes' -l vserv 147.101.224.39

Successful:

OpenSSH_2.9p1, SSH protocols 1.5/2.0, OpenSSL 0x0090581f
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Seeding random number generator
debug1: Rhosts Authentication disabled, originating port will not be
trusted.
debug1: restore_uid
debug1: ssh_connect: getuid 0 geteuid 0 anon 1
debug1: Connecting to 147.101.224.39 [147.101.224.39] port 22.
debug1: temporarily_use_uid: 0/0 (e=0)
debug1: restore_uid
debug1: temporarily_use_uid: 0/0 (e=0)
debug1: restore_uid
debug1: Connection established.
debug1: identity file /root/.ssh/identity type 0
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_rsa type 1
debug1: Remote protocol version 1.99, remote software version OpenSSH_2.9p1
debug1: match: OpenSSH_2.9p1 pat ^OpenSSH
Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_2.9p1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes256-cbc hmac-md5 none
debug1: kex: client->server aes256-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: dh_gen_key: priv key bits set: 262/512
debug1: bits set: 1006/2049
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host '147.101.224.39' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts2:2
debug1: bits set: 1063/2049
debug1: ssh_rsa_verify: signature correct
debug1: kex_derive_keys
debug1: newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: waiting for SSH2_MSG_NEWKEYS
debug1: newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: done: ssh_kex2.
debug1: send SSH2_MSG_SERVICE_REQUEST
debug1: service_accept: ssh-userauth
debug1: got SSH2_MSG_SERVICE_ACCEPT
debug1: authentications that can continue:
publickey,password,keyboard-interactive
debug1: next auth method to try is publickey
debug1: try privkey: /root/.ssh/id_dsa
debug1: try pubkey: /root/.ssh/id_rsa
debug1: input_userauth_pk_ok: pkalg ssh-rsa blen 149 lastkey 0x8112050 hint
2
debug1: read PEM private key done: type RSA
debug1: ssh-userauth2 successful: method publickey
debug1: fd 4 setting O_NONBLOCK
debug1: channel 0: new [client-session]
debug1: channel_new: 0
debug1: send channel open 0
debug1: Entering interactive session.
debug1: client_init id 0 arg 0
debug1: channel request 0: shell
debug1: channel 0: open confirm rwindow 0 rmax 16384
debug1: channel 0: read<=0 rfd 4 len 0
debug1: channel 0: read failed
debug1: channel 0: input open -> drain
debug1: channel 0: close_read
debug1: channel 0: input: no drain shortcut
debug1: channel 0: ibuf empty
debug1: channel 0: input drain -> closed
debug1: channel 0: send eof
debug1: channel 0: rcvd eof
debug1: channel 0: output open -> drain
  8:47am  up 49 day(s), 17:46,  2 users,  load average: 0.52, 0.49, 0.46
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: rcvd close
debug1: channel 0: obuf empty
debug1: channel 0: output drain -> closed
debug1: channel 0: close_write
debug1: channel 0: send close
debug1: channel 0: is dead
debug1: channel_free: channel 0: status: The following connections are
open:
  #0 client-session (t4 r0 i8/0 o128/0 fd -1/-1)

debug1: channel_free: channel 0: dettaching channel user
debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 0.1 seconds
debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.0
debug1: Exit status 0



Failure:

OpenSSH_2.9p1, SSH protocols 1.5/2.0, OpenSSL 0x0090581f
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Seeding random number generator
debug1: Rhosts Authentication disabled, originating port will not be
trusted.
debug1: restore_uid
debug1: ssh_connect: getuid 0 geteuid 0 anon 1
debug1: Connecting to 147.101.224.39 [147.101.224.39] port 22.
debug1: temporarily_use_uid: 0/0 (e=0)
debug1: restore_uid
debug1: temporarily_use_uid: 0/0 (e=0)
debug1: restore_uid
debug1: Connection established.
debug1: identity file /root/.ssh/identity type 0
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_rsa type 1
debug1: Remote protocol version 1.99, remote software version OpenSSH_2.9p1
debug1: match: OpenSSH_2.9p1 pat ^OpenSSH
Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_2.9p1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes256-cbc hmac-md5 none
debug1: kex: client->server aes256-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: dh_gen_key: priv key bits set: 260/512
debug1: bits set: 1042/2049
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host '147.101.224.39' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts2:2
debug1: bits set: 1054/2049
debug1: ssh_rsa_verify: signature correct
debug1: kex_derive_keys
debug1: newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: waiting for SSH2_MSG_NEWKEYS
debug1: newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: done: ssh_kex2.
debug1: send SSH2_MSG_SERVICE_REQUEST
debug1: service_accept: ssh-userauth
debug1: got SSH2_MSG_SERVICE_ACCEPT
debug1: authentications that can continue:
publickey,password,keyboard-interactive
debug1: next auth method to try is publickey
debug1: try privkey: /root/.ssh/id_dsa
debug1: try pubkey: /root/.ssh/id_rsa
debug1: input_userauth_pk_ok: pkalg ssh-rsa blen 149 lastkey 0x8112050 hint
2
debug1: read PEM private key done: type RSA
debug1: ssh-userauth2 successful: method publickey
debug1: fd 4 setting O_NONBLOCK
debug1: channel 0: new [client-session]
debug1: channel_new: 0
debug1: send channel open 0
debug1: Entering interactive session.
debug1: client_init id 0 arg 0
debug1: channel request 0: shell
debug1: channel 0: open confirm rwindow 0 rmax 16384
debug1: channel 0: read<=0 rfd 4 len 0
debug1: channel 0: read failed
debug1: channel 0: input open -> drain
debug1: channel 0: close_read
debug1: channel 0: input: no drain shortcut
debug1: channel 0: ibuf empty
debug1: channel 0: input drain -> closed
debug1: channel 0: send eof
debug1: channel 0: rcvd eof
debug1: channel 0: output open -> drain
  8:47am  up 49 day(s), 17:46,  2 users,  load average: 0.47, 0.48, 0.46
debug1: channel 0: obuf empty
debug1: channel 0: output drain -> closed
debug1: channel 0: close_write
debug1: channel 0: send close
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: rcvd close
debug1: channel 0: is dead
debug1: channel_free: channel 0: status: The following connections are
open:
  #0 client-session (t4 r0 i8/0 o128/0 fd -1/-1)

debug1: channel_free: channel 0: dettaching channel user