Hanging ssh

Tue Dec 19 03:58:00 EST 2000

OK, with openssh-SNAP-20001218 I still have the problem of a hanging ssh
when running (one specific) command on a server. On the server side,
there's a small suid C prog (/usr/dh/mgmt/HupServer) that runs system
"/usr/local/apache/bin/apachectl stop/start". Incidentally, apachectl is a
shell script, hence the need for a suid wrapper for non-root users... On
the client side, there's a shell script that removes the machine from load
balancing and then stops/starts the webserver by sshing to the server and
running the C prog there.

The problem doesn't arise if I run the HupServer program from the command
line, only from within the shell script that does the load balancing
enable/disable. In that script, the 'stop' command has no problems, only
'start'.

- I assumed that a fd was being left open, so I added:
	close(2); close(1); close(0); exit(0);
  to the C prog, but with no affect...
- Last I checked the -t switch had no affect...
- Hitting Ctrl-C on either debug run finishes the process up cleanly
- During the hanging there is no HupServer executable running on the
  server
- The problem was diminished, but not eliminated, by the prior patches to
  end the connection before the fds were closed.

Any ideas on how I can work around this? Would forking HupServer be of any
use?
								-Rob

The ssh dump:

SSH Version OpenSSH_2.3.0p2, protocol versions 1.5/2.0.
Compiled with SSL (0x0090581f).
debug: Reading configuration data /etc/ssh/ssh_config
debug: Applying options for *
debug: ssh_connect: getuid 500 geteuid 0 anon 0
debug: Connecting to dhumb400 [10.40.10.10] port 22.
debug: Allocated local port 1022.
debug: Connection established.
debug: identity file /home/prod/.ssh/identity type 0
debug: identity file /home/prod/.ssh/id_dsa type 3
debug: Remote protocol version 1.99, remote software version
OpenSSH_2.3.0p2
debug: no match: OpenSSH_2.3.0p2
Enabling compatibility mode for protocol 2.0
debug: Local version string SSH-2.0-OpenSSH_2.3.0p2
debug: Seeding random number generator
debug: send KEXINIT
debug: done
debug: wait KEXINIT
debug: got
kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug: got kexinit: ssh-dss,ssh-rsa
debug: got
kexinit: 3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes128-cbc,aes192-cbc,aes256-cbc,rijndael128-cbc,rijndael192-cbc,rijndael256-cbc,rijndael-cbc at lysator.liu.se
debug: got
kexinit: 3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes128-cbc,aes192-cbc,aes256-cbc,rijndael128-cbc,rijndael192-cbc,rijndael256-cbc,rijndael-cbc at lysator.liu.se
debug: got kexinit: hmac-sha1,hmac-md5,hmac-ripemd160 at openssh.com
debug: got kexinit: hmac-sha1,hmac-md5,hmac-ripemd160 at openssh.com
debug: got kexinit: none,zlib
debug: got kexinit: none,zlib
debug: got kexinit: 
debug: got kexinit: 
debug: first kex follow: 0 
debug: reserved: 0 
debug: done
debug: kex: server->client arcfour hmac-sha1 none
debug: kex: client->server arcfour hmac-sha1 none
debug: Sending SSH2_MSG_KEX_DH_GEX_REQUEST.
debug: Wait SSH2_MSG_KEX_DH_GEX_GROUP.
debug: Got SSH2_MSG_KEX_DH_GEX_GROUP.
debug: bits set: 1002/2049
debug: Sending SSH2_MSG_KEX_DH_GEX_INIT.
debug: Wait SSH2_MSG_KEX_DH_GEX_REPLY.
debug: Got SSH2_MSG_KEXDH_REPLY.
debug: Host 'dhumb400' is known and matches the RSA host key.
debug: bits set: 1034/2049
debug: ssh_rsa_verify: signature correct
debug: Wait SSH2_MSG_NEWKEYS.
debug: GOT SSH2_MSG_NEWKEYS.
debug: send SSH2_MSG_NEWKEYS.
debug: done: send SSH2_MSG_NEWKEYS.
debug: done: KEX2.
debug: send SSH2_MSG_SERVICE_REQUEST
debug: service_accept: ssh-userauth
debug: got SSH2_MSG_SERVICE_ACCEPT
debug: authentications that can continue: publickey,password
debug: next auth method to try is publickey
debug: try pubkey: /home/prod/.ssh/id_dsa
debug: read SSH2 private key done: name dsa w/o comment success 1
debug: sig size 20 20
debug: ssh-userauth2 successfull: method publickey
debug: channel 0: new [client-session]
debug: send channel open 0
debug: Entering interactive session.
debug: client_init id 0 arg 0
debug: Sending command: /usr/dh/mgmt/scripts/HupServer start
debug: channel 0: open confirm rwindow 0 rmax 16384
Environment:
  USER=prod
  LOGNAME=prod
  HOME=/usr/dh/home/prod
  PATH=/usr/dh/home/prod/bin:/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin
  MAIL=/var/mail/prod
  SHELL=/usr/local/bin/tcsh
  SSH_CLIENT=10.40.12.11 1022 22
/usr/local/apache/bin/apachectl start: httpd started
debug: client_input_channel_req: rtype exit-status reply 0
Connection to dhumb400 closed by remote host.
debug: Transferred: stdin 0, stdout 0, stderr 47 bytes in 138.8 seconds
debug: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.3
debug: Exit status 0

and the sshd dump:
debug1: sshd version OpenSSH_2.3.0p2
debug1: load_private_key_autodetect: type 0 RSA1
debug1: read SSH2 private key done: name dsa w/o comment success 1
debug1: load_private_key_autodetect: type 2 DSA
debug1: read SSH2 private key done: name rsa w/o comment success 1
debug1: load_private_key_autodetect: type 1 RSA
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
Generating 768 bit RSA key.
RSA key generation complete.
debug1: Server will not fork when running in debugging mode.
Connection from 10.40.12.11 port 1022
debug1: Client protocol version 2.0; client software version
OpenSSH_2.3.0p2
debug1: no match: OpenSSH_2.3.0p2
Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-1.99-OpenSSH_2.3.0p2
debug1: list_hostkey_types: ssh-dss,ssh-rsa
debug1: send KEXINIT
debug1: done
debug1: wait KEXINIT
debug1: got
kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug1: got kexinit: ssh-rsa,ssh-dss
debug1: got kexinit: arcfour,3des-cbc,blowfish-cbc,cast128-cbc
debug1: got kexinit: arcfour,3des-cbc,blowfish-cbc,cast128-cbc
debug1: got kexinit: hmac-sha1,hmac-md5,hmac-ripemd160 at openssh.com
debug1: got kexinit: hmac-sha1,hmac-md5,hmac-ripemd160 at openssh.com
debug1: got kexinit: none
debug1: got kexinit: none
debug1: got kexinit: 
debug1: got kexinit: 
debug1: first kex follow: 0 
debug1: reserved: 0 
debug1: done
debug1: kex: client->server arcfour hmac-sha1 none
debug1: kex: server->client arcfour hmac-sha1 none
debug1: Wait SSH2_MSG_KEX_DH_GEX_REQUEST.
debug1: Sending SSH2_MSG_KEX_DH_GEX_GROUP.
debug1: bits set: 1034/2049
debug1: Wait SSH2_MSG_KEX_DH_GEX_INIT.
debug1: bits set: 1002/2049
debug1: send SSH2_MSG_NEWKEYS.
debug1: done: send SSH2_MSG_NEWKEYS.
debug1: Wait SSH2_MSG_NEWKEYS.
debug1: GOT SSH2_MSG_NEWKEYS.
debug1: done: KEX2.
debug1: userauth-request for user prod service ssh-connection method none
debug1: attempt #1
debug1: Starting up PAM with username "prod"
Failed none for prod from 10.40.12.11 port 1022 ssh2
debug1: userauth-request for user prod service ssh-connection method
publickey
debug1: attempt #2
debug1: matching key found: file /usr/dh/home/prod/.ssh/authorized_keys2,
line 2
debug1: len 55 datafellows 0
debug1: ssh_dss_verify: signature correct
Could not reverse map address 10.40.12.11.
debug1: PAM setting rhost to "10.40.12.11"
Accepted publickey for prod from 10.40.12.11 port 1022 ssh2
debug1: Entering interactive session for SSH2.
debug1: server_init_dispatch_20
debug1: server_input_channel_open: ctype session rchan 0 win 65536 max
32768
debug1: input_session_request
debug1: channel 0: new [server-session]
debug1: session_new: init
debug1: session_new: session 0
debug1: session_open: channel 0
debug1: session_open: session 0: link with channel 0
debug1: server_input_channel_open: confirm session
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 channel 0 request exec reply
0
debug1: PAM establishing creds
debug1: fd 7 setting O_NONBLOCK
debug1: fd 7 IS O_NONBLOCK
debug1: fd 9 setting O_NONBLOCK
debug1: Received SIGCHLD.
debug1: tvp!=NULL kid 1 mili 100
debug1: session_by_pid: pid 34260
debug1: session_exit_message: session 0 channel 0 pid 34260
debug1: session_exit_message: release channel 0
debug1: channel 0: write failed
debug1: channel 0: output open -> closed
debug1: channel 0: close_write
debug1: session_free: session 0 pid 34260