openpty() and AIX

[Perry Smith]

I'm not on this mailing list so please make sure that I'm listed in  
any replies.

There seems to be a basic flaw in either AIX pty's or many Linux  
applications and sshd falls into this category.

sshd has a routine called openpty and it looks like ssh's version  
mimics the version in Linux.  (I'm not sure where openpty comes from  
-- I'm assuming Linux.)  The key to openpty is it opens both the  
slave and the master side of the pty.

sshd calls openpty before the fork of the child so the parent has  
both file descriptors open.  After the fork, it closes the file  
descriptor to the slave.  The problem is that if the child generates  
output and exits before the parent can close its file descriptor to  
the slave then the parent's file descriptor becomes the last file  
descriptor to the slave.  This implies that the close will go down to  
the device driver.

On AIX, if there is data to be read on the ptc (the master side) of  
the pty, then the slave device driver will sleep in the close  
routine.  It has been countless years since I have looked at the BSD  
code but I'm pretty sure this is what the BSD code did as well.  I  
can't say if I have ever looked at the AT&T 5.4 code.  AT&T prior to  
5.4 did not have pty's (as I recall).  To me, the BSD code defines  
how pty's work.  There are also various Posix rules but they change  
so often I can't keep up with them.  And, the behavior seems logical  
to me.  The slave side of a pty is suppose to look just like a tty  
and a tty will not close until all of the data has drained out  
(assuming various other things like DCD is still high, etc).

I see in at least some of the paths through the ssh code, the child  
opens the slave side of the pty -- which it needs to do in order to  
set up the process sessions.  But what I can not tell with the short  
time I've looked at the code if the code always opens the slave side  
(when there is a pty involved at all).  As far as I can tell, there  
are three basic paths to be concerned with:

1) just a normal ssh to a host
2) an ssh to a host that includes a command
3) an ssh to a host that includes a command and also uses the -t flag

If the child always opens the slave side, then it should be possible  
to close the slaves fd in the parent before the fork.  Utopia would  
be to never open the slave in the parent in the first place.  That is  
where I'm somewhat confused.  If all of the Linux applications like  
telnetd, rlogind, etc do the same thing, then it seems like they  
would all suffer from this problem.  But, I assume that is not the  
case so maybe a change to the AIX pty code could be done (but I don't  
see how -- which is why I'm asking).

So, my two basic questions are: 1) are the three paths through the  
code I listed above a complete list that I need to be looking at and  
2) can anyone comment on how the Linux pty's work?

Thank you,
Perry Smith ( pedz at easesoftware.com )
Ease Software, Inc. ( http://www.easesoftware.com )

Low cost SATA Disk Systems for IBMs p5, pSeries, and RS/6000 AIX systems