ssh 'connection reset by peer' problem since 5.8p1

[Oren Held]

On Thu, Feb 17, 2011 at 12:27:04AM +1100, Darren Tucker wrote:
> On 16/02/11 10:03 PM, Oren Held wrote:
> >On Wed, Feb 16, 2011 at 09:46:07PM +1100, Darren Tucker wrote:
> >>On 16/02/11 9:27 PM, Oren Held wrote:
> >>>Hi,
> >>>
> >>>As I haven't seen any related discussion here (or bug in bugzilla), I thought
> >>>it'll be good to do a heads-up for a recently introduced problem in 5.8p1 (or at
> >>>least after 5.5.x).
> >>>
> >>>Debian: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=612607
> >>>Ubuntu: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/708493
> >>>Arch Linux: https://bugs.archlinux.org/task/22897?project=1
> >>>
> >>>Any other information needed? Shall I file a bug in bugzilla?
> >>
> >>Can you reproduce the problem with an unmodified tarball from openssh.com?
> >
> >Yes. I've just compiled a "vanilla" 5.8p1, and same result:
> >
> >orenhe at orenhe-laptop:/tmp/openssh-5.8p1$ /opt/ssh/bin/ssh root at blade-8.ps
> >Read from socket failed: Connection reset by peer
> 
> When this happens, does it happen immediately or is there a delay
> before you get the "reset by peer" ?
Yes. It's very quick.

> Could you please collect both server and client side debug output?
> (note that you'll need to use sshd -ddde to make sure you get all
> the server output on stderr)

Attaching two files, one for the server's log and one for the client's.

One more note: the problem occurs when SSHing to *older* servers (I tried 4.3p2,
5.1p1). When SSHing to v5.8p1 it works smoothly.

-------------- next part --------------
debug2: load_server_config: filename /etc/ssh/sshd_config
debug2: load_server_config: done config len = 517
debug2: parse_server_config: config /etc/ssh/sshd_config len 517
debug1: sshd version OpenSSH_4.3p2
debug3: Not a RSA1 key file /etc/ssh/ssh_host_rsa_key.
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug3: Not a RSA1 key file /etc/ssh/ssh_host_dsa_key.
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-ddde'
debug1: rexec_argv[2]='-p'
debug1: rexec_argv[3]='2222'
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 2222 on ::.
Server listening on :: port 2222.
debug2: fd 4 setting O_NONBLOCK
debug1: Bind to port 2222 on 0.0.0.0.
Bind to port 2222 on 0.0.0.0 failed: Address already in use.


debug3: fd 4 is not O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 7 config len 517
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug1: rexec start in 4 out 4 newsock 4 pipe -1 sock 7
debug3: recv_rexec_state: entering fd = 5
debug3: ssh_msg_recv entering
debug3: recv_rexec_state: done
debug2: parse_server_config: config rexec len 517
debug1: sshd version OpenSSH_4.3p2
debug3: Not a RSA1 key file /etc/ssh/ssh_host_rsa_key.
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug3: Not a RSA1 key file /etc/ssh/ssh_host_dsa_key.
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: inetd sockets after dupping: 3, 3
debug3: Normalising mapped IPv4 in IPv6 address
Connection from 9.151.133.53 port 46862
debug1: Client protocol version 2.0; client software version OpenSSH_5.8
debug1: match: OpenSSH_5.8 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.3
debug2: fd 3 setting O_NONBLOCK
debug3: privsep user:group 74:74
debug1: permanently_set_uid: 74/74
debug1: list_hostkey_types: ssh-rsa,ssh-dss
debug1: SSH2_MSG_KEXINIT sent
debug2: Network child is on pid 1403
debug3: preauth child monitor started
debug3: mm_request_receive entering
Read from socket failed: Connection reset by peer
debug1: do_cleanup
debug3: PAM: sshpam_thread_cleanup entering
debug1: do_cleanup
debug3: PAM: sshpam_thread_cleanup entering

-------------- next part --------------
OpenSSH_5.8p1, OpenSSL 0.9.8o 01 Jun 2010
debug1: Reading configuration data /home/orenhe/.ssh/config
debug1: Reading configuration data /opt/ssh/etc/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to blade-8.ps [9.151.146.103] port 2222.
debug1: Connection established.
debug1: identity file /home/orenhe/.ssh/id_rsa type -1
debug1: identity file /home/orenhe/.ssh/id_rsa-cert type -1
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/orenhe/.ssh/id_dsa" as a RSA1 public key
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug2: key_type_from_name: unknown key type 'Proc-Type:'
debug3: key_read: missing keytype
debug2: key_type_from_name: unknown key type 'DEK-Info:'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /home/orenhe/.ssh/id_dsa type 2
debug1: identity file /home/orenhe/.ssh/id_dsa-cert type -1
debug1: identity file /home/orenhe/.ssh/id_ecdsa type -1
debug1: identity file /home/orenhe/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
debug1: match: OpenSSH_4.3 pat OpenSSH_4*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8
debug2: fd 3 setting O_NONBLOCK
debug3: put_host_port: [blade-8.ps]:2222
debug3: load_hostkeys: loading entries for host "[blade-8.ps]:2222" from file "/home/orenhe/.ssh/known_hosts"
debug3: load_hostkeys: loaded 0 keys
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01 at openssh.com,ecdsa-sha2-nistp384-cert-v01 at openssh.com,ecdsa-sha2-nistp521-cert-v01 at openssh.com,ssh-rsa-cert-v01 at openssh.com,ssh-dss-cert-v01 at openssh.com,ssh-rsa-cert-v00 at openssh.com,ssh-dss-cert-v00 at openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc at lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc at lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64 at openssh.com,hmac-ripemd160,hmac-ripemd160 at openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64 at openssh.com,hmac-ripemd160,hmac-ripemd160 at openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib at openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib at openssh.com,zlib
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc at lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc at lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160 at openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160 at openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib at openssh.com
debug2: kex_parse_kexinit: none,zlib at openssh.com
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
Write failed: Connection reset by peer