Hanging ssh

Wed Dec 20 02:27:47 EST 2000

Your comments got me thinking; I tried my situation using Protocol 1
(forced by sshd) and no problems so far... the repeatable problem using
snaps later than 20001114 (I believe the last one to contain the
patch) has gone away, and it survived the night without leaving a hung
process around (which even 20001114 did).

Is this of any use? Is there anything else I can do to help track this
down? I'd rather not have to convert to ssh1... <shudder>
								-Rob

On Mon, 18 Dec 2000 mouring at etoh.eviladmin.org wrote:

> 
> What platform?  I know Linux has such an issue.  The patch to resolve the
> issue caused scp to not complete it's transfers under meduim loads so
> it was removed.  I have/had (somewhere) a hack to work around scp for
> protocol 1. That did not SEEM to cause problems.. But I was not happy
> with it. 
> 
> I've not been able to verify but does the old 1.2.16 source tree from
> ssh.com produce the same results?  There are some major rewrites in the
> lower level channel support that may be the problem.  
> 
> - Ben
> 
> On Mon, 18 Dec 2000, Rachit Siamwalla wrote:
> 
> > does this happen if you test this through rsh?
> > 
> > If so, i've had a similar problem (unfortunately no good solution
> > though). The way i "solved" the problem is writing my own shellscript
> > that had the same effect. apachectl is probably a shell script too. try
> > cutting out only what is needed and narrow the problem down from there.
> > 
> > -rchit
> > 
> > 
> > Rob Hagopian wrote:
> > > 
> > > OK, with openssh-SNAP-20001218 I still have the problem of a hanging ssh
> > > when running (one specific) command on a server. On the server side,
> > > there's a small suid C prog (/usr/dh/mgmt/HupServer) that runs system
> > > "/usr/local/apache/bin/apachectl stop/start". Incidentally, apachectl is a
> > > shell script, hence the need for a suid wrapper for non-root users... On
> > > the client side, there's a shell script that removes the machine from load
> > > balancing and then stops/starts the webserver by sshing to the server and
> > > running the C prog there.
> > > 
> > > The problem doesn't arise if I run the HupServer program from the command
> > > line, only from within the shell script that does the load balancing
> > > enable/disable. In that script, the 'stop' command has no problems, only
> > > 'start'.
> > > 
> > > - I assumed that a fd was being left open, so I added:
> > >         close(2); close(1); close(0); exit(0);
> > >   to the C prog, but with no affect...
> > > - Last I checked the -t switch had no affect...
> > > - Hitting Ctrl-C on either debug run finishes the process up cleanly
> > > - During the hanging there is no HupServer executable running on the
> > >   server
> > > - The problem was diminished, but not eliminated, by the prior patches to
> > >   end the connection before the fds were closed.
> > > 
> > > Any ideas on how I can work around this? Would forking HupServer be of any
> > > use?
> > >                                                                 -Rob
> > > 
> > > The ssh dump:
> > > 
> > > SSH Version OpenSSH_2.3.0p2, protocol versions 1.5/2.0.
> > > Compiled with SSL (0x0090581f).
> > > debug: Reading configuration data /etc/ssh/ssh_config
> > > debug: Applying options for *
> > > debug: ssh_connect: getuid 500 geteuid 0 anon 0
> > > debug: Connecting to dhumb400 [10.40.10.10] port 22.
> > > debug: Allocated local port 1022.
> > > debug: Connection established.
> > > debug: identity file /home/prod/.ssh/identity type 0
> > > debug: identity file /home/prod/.ssh/id_dsa type 3
> > > debug: Remote protocol version 1.99, remote software version
> > > OpenSSH_2.3.0p2
> > > debug: no match: OpenSSH_2.3.0p2
> > > Enabling compatibility mode for protocol 2.0
> > > debug: Local version string SSH-2.0-OpenSSH_2.3.0p2
> > > debug: Seeding random number generator
> > > debug: send KEXINIT
> > > debug: done
> > > debug: wait KEXINIT
> > > debug: got
> > > kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
> > > debug: got kexinit: ssh-dss,ssh-rsa
> > > debug: got
> > > kexinit: 3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes128-cbc,aes192-cbc,aes256-cbc,rijndael128-cbc,rijndael192-cbc,rijndael256-cbc,rijndael-cbc at lysator.liu.se
> > > debug: got
> > > kexinit: 3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes128-cbc,aes192-cbc,aes256-cbc,rijndael128-cbc,rijndael192-cbc,rijndael256-cbc,rijndael-cbc at lysator.liu.se
> > > debug: got kexinit: hmac-sha1,hmac-md5,hmac-ripemd160 at openssh.com
> > > debug: got kexinit: hmac-sha1,hmac-md5,hmac-ripemd160 at openssh.com
> > > debug: got kexinit: none,zlib
> > > debug: got kexinit: none,zlib
> > > debug: got kexinit:
> > > debug: got kexinit:
> > > debug: first kex follow: 0
> > > debug: reserved: 0
> > > debug: done
> > > debug: kex: server->client arcfour hmac-sha1 none
> > > debug: kex: client->server arcfour hmac-sha1 none
> > > debug: Sending SSH2_MSG_KEX_DH_GEX_REQUEST.
> > > debug: Wait SSH2_MSG_KEX_DH_GEX_GROUP.
> > > debug: Got SSH2_MSG_KEX_DH_GEX_GROUP.
> > > debug: bits set: 1002/2049
> > > debug: Sending SSH2_MSG_KEX_DH_GEX_INIT.
> > > debug: Wait SSH2_MSG_KEX_DH_GEX_REPLY.
> > > debug: Got SSH2_MSG_KEXDH_REPLY.
> > > debug: Host 'dhumb400' is known and matches the RSA host key.
> > > debug: bits set: 1034/2049
> > > debug: ssh_rsa_verify: signature correct
> > > debug: Wait SSH2_MSG_NEWKEYS.
> > > debug: GOT SSH2_MSG_NEWKEYS.
> > > debug: send SSH2_MSG_NEWKEYS.
> > > debug: done: send SSH2_MSG_NEWKEYS.
> > > debug: done: KEX2.
> > > debug: send SSH2_MSG_SERVICE_REQUEST
> > > debug: service_accept: ssh-userauth
> > > debug: got SSH2_MSG_SERVICE_ACCEPT
> > > debug: authentications that can continue: publickey,password
> > > debug: next auth method to try is publickey
> > > debug: try pubkey: /home/prod/.ssh/id_dsa
> > > debug: read SSH2 private key done: name dsa w/o comment success 1
> > > debug: sig size 20 20
> > > debug: ssh-userauth2 successfull: method publickey
> > > debug: channel 0: new [client-session]
> > > debug: send channel open 0
> > > debug: Entering interactive session.
> > > debug: client_init id 0 arg 0
> > > debug: Sending command: /usr/dh/mgmt/scripts/HupServer start
> > > debug: channel 0: open confirm rwindow 0 rmax 16384
> > > Environment:
> > >   USER=prod
> > >   LOGNAME=prod
> > >   HOME=/usr/dh/home/prod
> > >   PATH=/usr/dh/home/prod/bin:/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin
> > >   MAIL=/var/mail/prod
> > >   SHELL=/usr/local/bin/tcsh
> > >   SSH_CLIENT=10.40.12.11 1022 22
> > > /usr/local/apache/bin/apachectl start: httpd started
> > > debug: client_input_channel_req: rtype exit-status reply 0
> > > Connection to dhumb400 closed by remote host.
> > > debug: Transferred: stdin 0, stdout 0, stderr 47 bytes in 138.8 seconds
> > > debug: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.3
> > > debug: Exit status 0
> > > 
> > > and the sshd dump:
> > > debug1: sshd version OpenSSH_2.3.0p2
> > > debug1: load_private_key_autodetect: type 0 RSA1
> > > debug1: read SSH2 private key done: name dsa w/o comment success 1
> > > debug1: load_private_key_autodetect: type 2 DSA
> > > debug1: read SSH2 private key done: name rsa w/o comment success 1
> > > debug1: load_private_key_autodetect: type 1 RSA
> > > debug1: Bind to port 22 on 0.0.0.0.
> > > Server listening on 0.0.0.0 port 22.
> > > Generating 768 bit RSA key.
> > > RSA key generation complete.
> > > debug1: Server will not fork when running in debugging mode.
> > > Connection from 10.40.12.11 port 1022
> > > debug1: Client protocol version 2.0; client software version
> > > OpenSSH_2.3.0p2
> > > debug1: no match: OpenSSH_2.3.0p2
> > > Enabling compatibility mode for protocol 2.0
> > > debug1: Local version string SSH-1.99-OpenSSH_2.3.0p2
> > > debug1: list_hostkey_types: ssh-dss,ssh-rsa
> > > debug1: send KEXINIT
> > > debug1: done
> > > debug1: wait KEXINIT
> > > debug1: got
> > > kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
> > > debug1: got kexinit: ssh-rsa,ssh-dss
> > > debug1: got kexinit: arcfour,3des-cbc,blowfish-cbc,cast128-cbc
> > > debug1: got kexinit: arcfour,3des-cbc,blowfish-cbc,cast128-cbc
> > > debug1: got kexinit: hmac-sha1,hmac-md5,hmac-ripemd160 at openssh.com
> > > debug1: got kexinit: hmac-sha1,hmac-md5,hmac-ripemd160 at openssh.com
> > > debug1: got kexinit: none
> > > debug1: got kexinit: none
> > > debug1: got kexinit:
> > > debug1: got kexinit:
> > > debug1: first kex follow: 0
> > > debug1: reserved: 0
> > > debug1: done
> > > debug1: kex: client->server arcfour hmac-sha1 none
> > > debug1: kex: server->client arcfour hmac-sha1 none
> > > debug1: Wait SSH2_MSG_KEX_DH_GEX_REQUEST.
> > > debug1: Sending SSH2_MSG_KEX_DH_GEX_GROUP.
> > > debug1: bits set: 1034/2049
> > > debug1: Wait SSH2_MSG_KEX_DH_GEX_INIT.
> > > debug1: bits set: 1002/2049
> > > debug1: send SSH2_MSG_NEWKEYS.
> > > debug1: done: send SSH2_MSG_NEWKEYS.
> > > debug1: Wait SSH2_MSG_NEWKEYS.
> > > debug1: GOT SSH2_MSG_NEWKEYS.
> > > debug1: done: KEX2.
> > > debug1: userauth-request for user prod service ssh-connection method none
> > > debug1: attempt #1
> > > debug1: Starting up PAM with username "prod"
> > > Failed none for prod from 10.40.12.11 port 1022 ssh2
> > > debug1: userauth-request for user prod service ssh-connection method
> > > publickey
> > > debug1: attempt #2
> > > debug1: matching key found: file /usr/dh/home/prod/.ssh/authorized_keys2,
> > > line 2
> > > debug1: len 55 datafellows 0
> > > debug1: ssh_dss_verify: signature correct
> > > Could not reverse map address 10.40.12.11.
> > > debug1: PAM setting rhost to "10.40.12.11"
> > > Accepted publickey for prod from 10.40.12.11 port 1022 ssh2
> > > debug1: Entering interactive session for SSH2.
> > > debug1: server_init_dispatch_20
> > > debug1: server_input_channel_open: ctype session rchan 0 win 65536 max
> > > 32768
> > > debug1: input_session_request
> > > debug1: channel 0: new [server-session]
> > > debug1: session_new: init
> > > debug1: session_new: session 0
> > > debug1: session_open: channel 0
> > > debug1: session_open: session 0: link with channel 0
> > > debug1: server_input_channel_open: confirm session
> > > debug1: session_by_channel: session 0 channel 0
> > > debug1: session_input_channel_req: session 0 channel 0 request exec reply
> > > 0
> > > debug1: PAM establishing creds
> > > debug1: fd 7 setting O_NONBLOCK
> > > debug1: fd 7 IS O_NONBLOCK
> > > debug1: fd 9 setting O_NONBLOCK
> > > debug1: Received SIGCHLD.
> > > debug1: tvp!=NULL kid 1 mili 100
> > > debug1: session_by_pid: pid 34260
> > > debug1: session_exit_message: session 0 channel 0 pid 34260
> > > debug1: session_exit_message: release channel 0
> > > debug1: channel 0: write failed
> > > debug1: channel 0: output open -> closed
> > > debug1: channel 0: close_write
> > > debug1: session_free: session 0 pid 34260
> > 
> 