Linux glibc 2.1 openpty() and /dev/ptmx

[Marek Michalkiewicz]

Hello,

looking at the pty handling in OpenSSH 2.3.0p1 (hasn't changed much in
CVS, as far as I can tell after a quick look at it), I can see that if
the system provides both /dev/ptmx and openpty() types of pty interface,
the latter is preferred.  This is the case on Linux with glibc 2.1.3
and most likely later versions too.  However, openpty() is documented
to be dangerous - quote from the glibc manual:

     *Warning:* Using the `openpty' function with NAME not set to
     `NULL' is *very dangerous* because it provides no protection
     against overflowing the string NAME.  You should use the `ttyname'
     function on the file descriptor returned in *SLAVE to find out the
     file name of the slave pseudo-terminal device instead.

Now, OpenSSH uses a 64-byte buffer for the tty name.  I don't know if
this is really exploitable, but the warning sounds serious.  Perhaps
openpty() in OpenBSD is guaranteed to never overflow such a buffer, but
that assumption might not hold for glibc?

But, glibc 2.1 also supports the standard Unix98 pty interface (/dev/ptmx),
which avoids the issue completely, and the modified OpenSSH (using exactly
the same pty code as for Solaris) appears to work fine in my tests, done
on a Debian "potato" system.

Are there any known problems with using /dev/ptmx instead of openpty()
if both interfaces are available?  Currently the former is completely
disabled for Linux by configure, and even if it isn't, if HAVE_OPENPTY
is defined, this causes HAVE_DEV_PTMX to be undefined.  Why?  (There is
a comment saying that pty allocated with _getpty gets broken, but that
is no problem in this case, as the pty is allocated by opening /dev/ptmx
and not _getpty.)

One more (minor) thing I noticed: the tests for no_libsocket and no_libnsl
in configure.in are swapped.  Systems that define one define the other too
so everything still worked right...

if test -z "$no_libsocket" ; then
	AC_CHECK_LIB(nsl, yp_match, , )
fi
if test -z "$no_libnsl" ; then
	AC_CHECK_LIB(socket, main, , )
fi

Thanks, and keep up the good work!
Marek