SCO OpenServer 5.0.5, issues with MAXPATHLEN and tty devices

[Aran Cox]

These issues affected previous 2. versions of openssh and still 
affects openssh-2.1.1p1.  The environment is SCO OS 5.0.5 with 
the SCO development environment (not gcc, gmake, etc.)

There are two issues, the first is a compilation problem, 
and the second is a run-time problem.

The first problem is simply that MAXPATHLEN is not defined.  I 
found it to be defined in /usr/include/sys/fs/dtfilsys.h but I 
am not sure if including this file is the correct thing to do 
or not.  I got around this by simply defining it as 1024 in 
defines.h. 

The second problem is related the tty devices chosen by the 
sshd when initiating a new connection:


If HAVE_DEV_PTMX is defined, code in pty.c (function pty_alloc) is
used that seems to be designed for Solaris 2.X.  The header above the
code is 
/*
 * This code is used e.g. on Solaris 2.x.  (Note that Solaris 2.3
 * also has bsd-style ptys, but they simply do not work.)
 */

It tries to use device names like /dev/pts000 and the code in
pty_make_controlling_tty to fail.  Specifically this code fails:

/* Verify that we now have a controlling tty. */
fd = open("/dev/tty", O_WRONLY);
if (fd < 0)
   error("open /dev/tty failed - could not set controlling tty: %.100s",
   strerror(errno));
else {
   close(fd);
}

Causing this message to be generated by the sshd when run with the -d
option:

error: open /dev/tty failed - could not set controlling tty: No such device or address

If I alter the config.h line that defines HAVE_DEV_PTMX to:
#undef HAVE_DEV_PTMX
then it compiles with code that seems to work exactly as expected,
choosing tty device names like /dev/ttyp8.

The above error message does indicate a real problem, not just
a potential one.
At the very least it prevents me from running resize which makes
using the terminal very tricky.


Both of these issues seem really to be problems with the configure
script.  (configure should find MAXPATHLEN for you, and should
know that using /dev/ptsX on SCO OS doesn't work) 
I'm no autoconf wizard but I will try to fix these issues 
and submit a patch, unless someone else who is more familiar
with the code could do it quicker ;)