Help with ssh -A, screen, ssh -a, detach, logout

[raf]

Bob Proulx wrote:

> raf wrote:
> > I've noticed some ssh behaviour that I wish didn't
> > happen. I was wondering if someone can explain how I
> > can stop it from happening, or explain why it's
> > unavoidable.
> 
> The problem you are running into is due to using a live network
> connection from the remote system back to your local system's
> ssh-agent.  If you weren't doing that then you would not run into this
> problem.  If you want to continue to do this then you will need to
> work around the problem.
> 
> > If I ssh-with-agent-forwarding from one host to a
> > second host, and on the second host use something like
> 
> That's the live network connection that you are setting up.
> 
> > nohup/screen/tmux/daemon, and from within that new
> 
> And those are long running environments.  Meaning that if you break
> the connection to the ssh-agent then it is possible to create
> situations where things running in the the above environments want to
> keep talking to the ssh-agent.

After the initial authentication on host2 to host3, there should
be no need for further access to the agent (because that ssh
was invoked with the -a option).

> > process session, start a long-running command via
> > ssh-without-agent-forwarding on a third host, I would
> > expect to be able to (e.g.) detach from the screen
> > session and log out of the second host, but my shell
> 
> In order to be able to expect to detach from the session and log out
> then the environment must not keep any resources open.  And by
> resources here it is the file descriptor that is connected to the
> network socket that is connected to the ssh-agent.  That's open.
> Meaning a non-zero reference count.  Meaning that it does not get
> closed.  Meaning that ssh is going to keep the connection open.
> "Because someone is using it."

My question is about why sshd on host2 keeps that file descriptor
open, or why the ssh on host2 (to host3) keeps it open after
authentication to host3 (after which it is no longer needed).

> > prompt on the first host doesn't come back and even
> > Ctrl-C won't break the connection between ssh on the
> > first host and sshd on the second host. I have to close
> > the xterm window that the shell and ssh are running in.
> 
> You could also use "Enter ~ ." to forcibly close the connection too.
> That is a useful command sequence.  The ~ is the escape character and
> is recognized at the beginning of a line.  See the manual in the
> section under "ESCAPE CHARACTERS" for the full description.

Thanks but that doesn't work, at least not after
logging out of host2 and waiting for the prompt on
host1.

Ah, but it does work after detaching from screen.
Thanks. That's helpful, but I need to know in advance,
when the ssh connection from host1 to host2 isn't going
to terminate cleanly. Then I can use the escape sequence.
If I can't tell in advance, and I try to log out, it
doesn't help. But thanks for the suggestion.

> > If I don't do that, the shell prompt doesn't come back
> > until the long-running command on the third host has
> > completed.
> 
> Correct.  That is the correct behavior.  The long running command on
> the remove is holding the file open.  Non-zero reference count.  When
> the process exits then it closes the file.  Which closes the network
> connection.  Which allows ssh to exit.

I think that if the ssh -a from host2 to host3 closed
its connection to the agent once it had finished with
it, then so would the sshd process on host2 (maybe),
and that would allow ssh from host1 to host2 to exit
normally. I'm only theorizing about that. I haven't
looked at the code yet.

> > To see what I mean:
> > 
> >   - on host1: Have ssh-agent running with an identity loaded
> >   - on host1: "xterm &" (start an xterm on similar)
> 
> All good.
> 
> >   - on host1 in xterm: "ssh -A host2" (ssh-with-agent-forwarding to host2)
> 
> At this point my personal opinion is that we should pause and think
> about why -A might be wanted here.  I realize the option exists.  I

It is wanted there because I need to ssh from host2 to host3
and that connection needs to be authenticated and I don't want
keys on host2 (and all the other hosts I use).

> realize that many people use this option a lot.  But personally I
> almost never use that option.  I don't need to use that option.  That
> option is just a door that opens to a room filled with a lot of
> security layer related questions.  Which might be okay.  Or might be
> a door I don't want to open.
> 
> Why are you using -A there?  Do you really need to use it?  That would
> be a good discussion point.  Because I don't ever have a need for it.
> Using it means one must trust the remote system not to be malicious.
> (Which it mostly will not be.  But it is possible.)  But mostly
> because the live network connection it sets up is then required to
> stay available for the full lifecycle.  As you have found out.  It
> creates entanglements.  It's messy.

Can anyone explain exactly why that connection *is*
required to stay available for the full lifecycle. As
far as I can tell, it only needs to stay available
until the authentication for the ssh connection to
host3 is complete. After the authentication, there is
no need for it that I am aware of (because the agent
is not being forwarded to host3).

> >   - on host2: "screen" (start a screen session)
> 
> And this sets up a pitfall that might or might not be fallen into.  In
> the screen environment for every shell started within it will be the
> environment variables from the ssh connection.  You will probably see
> something like this example.
> 
>   rwp at madness:~$ env | grep SSH
>   SSH_AUTH_SOCK=/tmp/ssh-YsdgP0Eexk/agent.14641
>   SSH_CONNECTION=192.168.230.119 44148 192.168.230.123 22
>   SSH_CLIENT=192.168.230.119 44148 22
>   SSH_TTY=/dev/pts/4
> 
> The problem is the SSH_AUTH_SOCK which is setting up the connectivity
> to the ssh-agent on your originating client.  If you avoid that then
> you avoid the problem.  I daresay the easiest way to avoid it is to
> avoid the -A option.  But if you must use it then when setting up
> screen you can remove it from the environment.
> 
>   env -u SSH_AUTH_SOCK screen

No. That doesn't solve the problem. It prevents the ssh
to host3 from having access to the agent which it needs
initially, so as to be able to authenticate the
connection. The subsequent use of ssh -a means that the
access to the agent will not be needed after that point
in time (as far as I can tell).

> Here is am using 'env' to unset the variable from the environment.
> And also 'env' is an idiom for a canonical way to set or clear
> environment variables regardless of the command line shell that anyone
> might be using.  Because bash, ksh, zsh, csh, and all of those have
> slightly different syntax.  But invoking 'env' this way would be
> identical in all of them.  Which makes it easiest for me to suggest
> using env in this way and knowing that it will work regardless of the
> user shell environment.  Isn't that pretty nice? :-)
> 
> >   - on host2 in screen: "ssh -a host3 sleep 60" (long-running cmd on host3)
> 
> And here you are using -a to *avoid* the ssh-agent that was set up
> with the -A in the originating invocation.  Layers and layers!  If the
> originating -A was removed then this -a could be removed.  Simplify!

Except that the originating -A is needed to be able to authenticate
the connection from host2 to host3.

> >   - on host2 in screen: Ctrl-a d (detach from the screen session)
> 
> But it really can't!  Because of the live long running network
> connection to the ssh-agent.  "The cake is a lie!"

Yes it can. Detaching from screen works fine.
It's only the logging out of host2 afterwards
that is a problem.

> >   - on host2: Ctrl-d (log out of host2)
> 
> This is not quite a "log out of host2".  This is an "exit the command
> line shell running on host2".  The difference is important in this
> specific case.  Because the command line shell will exit.  But the
> command line shell is running under sshd on the remote host.  And that
> is talking to the ssh on the local host.  And as described the remote
> sshd is going to need to keep running waiting for the live network
> connection to your ssh-agent to close.

Yes, I know. But what I don't know is why that sshd
process on host2 still has the additional connection to
the agent socket that it created when the ssh to host3
started. I can't help thinking that if the ssh to host3
closed its connection when it no longer needed it, the
sshd process would probably also close its
corresponding connection (just guessing), and then
there would no longer be any ssh channel that needed to
stay open and prevent me from logging out of host2.

> >   - on host1: wait a long time for the shell prompt to appear or close xterm
> 
> Right.  As it should be doing.  Due to the use of -A.

No. If I use ssh -A to log into host2 and then log out
again, this problem does not exist. The problem is not
the -A option. The problem is only when an additional
connection to the agent is created and not closed when
it is no longer needed.

> > In other words, I want the agent to be forwarded to
> > host2, so that I can then ssh from there to host3, but
> > I don't want the agent to be forwarded to host3 because
> > it's not needed there. Note that my real command was
> > rsync so both host2 and host3 were involved.
> 
> That is the critical point.  I've written a bunch here already.  With
> a lot of opinions embedded! :-)  So I will more or less pause for a
> moment here for reflection.  Because everything becomes interconnected
> and understanding those interconnections will make working with things
> all make sense.  "Know how the system works.  Know how to work the
> system." :-)
> 
> > My hypothesis is that agent forwarding has something to
> > do with why the connection between host1 and host2
> > isn't cleanly closed.
> 
> And I believe your hypothesis to be a correct one.
> 
> > Any suggestions?
> 
> [...keychain snip...]

> Hope this helps! :-)

It doesn't help much but many thanks for trying.

> Bob

cheers,
raf