Strange problem from "identical" hosts

[Ladner, Eric (CLAD)]

Long post.. sorry.

Ok.. I've got three systems, all running openssh-3.0.2p1.  As a matter 
of fact, they were installed from the same built tree, so I know they are
the same.

Here's the deal.  I've got three systems, call them source1, source2 and 
target.  All are HP-UX 11.0 systems installed from the same tree.

Source1 and source2 both have thier root rsa pub keys in target's auth 
keys file.

If I ssh over from source1, everything works great.  If I ssh over from 
source2, it asks me for root's password.

Here's a debug list of the ssh from source1 and source2 with the 
differences indicated by a leading >:

Any clues?

Source1 (the one that works)

   # ssh-agent /bin/ksh
   # ssh-add /root/.ssh/id_rsa
   Enter passphrase for /root/.ssh/id_rsa:
   Identity added: /root/.ssh/id_rsa (/root/.ssh/id_rsa)
   # ssh -v target
   OpenSSH_3.0.2p1, SSH protocols 1.5/2.0, OpenSSL 0x0090601f
   debug1: Reading configuration data /etc/opt/ssh-3.0/ssh_config
   debug1: Applying options for *
   debug1: Seeding random number generator
   debug1: Rhosts Authentication disabled, originating port will not be
trusted.
   debug1: restore_uid
   debug1: ssh_connect: getuid 0 geteuid 0 anon 1
   debug1: Connecting to target [10.0.0.253] port 22.
   debug1: temporarily_use_uid: 0/3 (e=0)
   debug1: restore_uid
   debug1: temporarily_use_uid: 0/3 (e=0)
   debug1: restore_uid
   debug1: Connection established.
   debug1: read PEM private key done: type DSA
   debug1: read PEM private key done: type RSA
   debug1: identity file /root/.ssh/id_dsa type -1
   debug1: identity file /root/.ssh/id_rsa type 1
>  debug1: identity file /root/.ssh/identity type 0
   debug1: Remote protocol version 2.0, remote software version
OpenSSH_3.0.2p1
   debug1: match: OpenSSH_3.0.2p1 pat ^OpenSSH
   Enabling compatibility mode for protocol 2.0
   debug1: Local version string SSH-2.0-OpenSSH_3.0.2p1
   debug1: SSH2_MSG_KEXINIT sent
   debug1: SSH2_MSG_KEXINIT received
   debug1: kex: server->client aes128-cbc hmac-md5 none
   debug1: kex: client->server aes128-cbc hmac-md5 none
   debug1: SSH2_MSG_KEX_DH_GEX_REQUEST sent
   debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
>  debug1: dh_gen_key: priv key bits set: 114/256
>  debug1: bits set: 1558/3191
   debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
   debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
   debug1: Host 'target' is known and matches the RSA host key.
   debug1: Found key in /root/.ssh/known_hosts:1
>  debug1: bits set: 1575/3191
   debug1: ssh_rsa_verify: signature correct
   debug1: kex_derive_keys
   debug1: newkeys: mode 1
   debug1: SSH2_MSG_NEWKEYS sent
   debug1: waiting for SSH2_MSG_NEWKEYS
   debug1: newkeys: mode 0
   debug1: SSH2_MSG_NEWKEYS received
   debug1: done: ssh_kex2.
   debug1: send SSH2_MSG_SERVICE_REQUEST
   debug1: service_accept: ssh-userauth
   debug1: got SSH2_MSG_SERVICE_ACCEPT
   debug1: authentications that can continue:
publickey,password,keyboard-interactive
   debug1: next auth method to try is publickey
   debug1: userauth_pubkey_agent: testing agent key /root/.ssh/id_rsa
The big difference starts here.
>  debug1: input_userauth_pk_ok: pkalg ssh-rsa blen 149 lastkey 40028aa8
hint -1
>  debug1: ssh-userauth2 successful: method publickey
>  debug1: channel 0: new [client-session]
>  debug1: send channel open 0
>  debug1: Entering interactive session.
>  debug1: ssh_session2_setup: id 0
>  debug1: Requesting authentication agent forwarding.
>  debug1: channel request 0: shell
>  debug1: channel 0: open confirm rwindow 0 rmax 16384
>  Last login: Thu Jan  3 16:12:22 2002 from source1
.. Login continues here..

Source2 (the one that doesn't work)

   # ssh-agent /bin/ksh
   # ssh-add /root/.ssh/id_rsa
   Enter passphrase for /root/.ssh/id_rsa:
   Identity added: /root/.ssh/id_rsa (/root/.ssh/id_rsa)
   # ssh -v target
   OpenSSH_3.0.2p1, SSH protocols 1.5/2.0, OpenSSL 0x0090601f
   debug1: Reading configuration data /etc/opt/ssh-3.0/ssh_config
   debug1: Applying options for *
   debug1: Seeding random number generator
   debug1: Rhosts Authentication disabled, originating port will not be
trusted.
   debug1: restore_uid
   debug1: ssh_connect: getuid 0 geteuid 0 anon 1
   debug1: Connecting to target [10.0.0.253] port 22.
   debug1: temporarily_use_uid: 0/3 (e=0)
   debug1: restore_uid
   debug1: temporarily_use_uid: 0/3 (e=0)
   debug1: restore_uid
   debug1: Connection established.
   debug1: read PEM private key done: type DSA
   debug1: read PEM private key done: type RSA
   debug1: identity file /root/.ssh/id_dsa type -1
   debug1: identity file /root/.ssh/id_rsa type 1
>  debug1: identity file /root/.ssh/identity type -1
   debug1: Remote protocol version 2.0, remote software version
OpenSSH_3.0.2p1
   debug1: match: OpenSSH_3.0.2p1 pat ^OpenSSH
   Enabling compatibility mode for protocol 2.0
   debug1: Local version string SSH-2.0-OpenSSH_3.0.2p1
   debug1: SSH2_MSG_KEXINIT sent
   debug1: SSH2_MSG_KEXINIT received
   debug1: kex: server->client aes128-cbc hmac-md5 none
   debug1: kex: client->server aes128-cbc hmac-md5 none
   debug1: SSH2_MSG_KEX_DH_GEX_REQUEST sent
   debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
>  debug1: dh_gen_key: priv key bits set: 111/256
>  debug1: bits set: 1623/3191
   debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
   debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
   debug1: Host 'target' is known and matches the RSA host key.
   debug1: Found key in /root/.ssh/known_hosts:1
>  debug1: bits set: 1597/3191
   debug1: ssh_rsa_verify: signature correct
   debug1: kex_derive_keys
   debug1: newkeys: mode 1
   debug1: SSH2_MSG_NEWKEYS sent
   debug1: waiting for SSH2_MSG_NEWKEYS
   debug1: newkeys: mode 0
   debug1: SSH2_MSG_NEWKEYS received
   debug1: done: ssh_kex2.
   debug1: send SSH2_MSG_SERVICE_REQUEST
   debug1: service_accept: ssh-userauth
   debug1: got SSH2_MSG_SERVICE_ACCEPT
   debug1: authentications that can continue:
publickey,password,keyboard-interactive
   debug1: next auth method to try is publickey
   debug1: userauth_pubkey_agent: testing agent key /root/.ssh/id_rsa
The big difference starts here.
>  debug1: authentications that can continue:
publickey,password,keyboard-interactive
>  debug1: try privkey: /root/.ssh/id_dsa
>  debug1: try pubkey: /root/.ssh/id_rsa
>  debug1: authentications that can continue:
publickey,password,keyboard-interactive
>  debug1: try privkey: /root/.ssh/identity
>  debug1: next auth method to try is keyboard-interactive
>  debug1: authentications that can continue:
publickey,password,keyboard-interactive
>  debug1: next auth method to try is password
>  root at target's password: