OpenSSH 2.3.0p1: HP-UX 11.00 64-bit

[Quentin Bracken]

Kevin, HP-UX 11 does use /dev/ptmx and HAVE_DEV_PTMX is defined when
configure runs on an HP-UX 11.00 64-bit system.  The suggested patch would
still result in a broken OpenSSH under 64-bit HP-UX 11.

I believe that the problem is actually in the vhangup(2) implementation in
the HP-UX 11.00 64-bit kernel.  It runs without fail on HP-UX 11.00 32-bit
and 10.20.  Reviewing the vmunix error message:

    System call 76 (vhangup) was called in a kernel where the type of at
least one of its arguments is currently unspecified.

leads me to believe that there is a deficiency in the vhangup implementation
in the HP-UX 64-bit kernel itself.  It is also worth mentioning that the
vhangup(2) system call is completely undocumented in any man page on 11.00 or
10.20 which I find somewhat suspicious.

Kevin Steves wrote:

> On Tue, 13 Mar 2001, Quentin Bracken wrote:
> : I have encountered a problem with using OpenSSH 2.3.0p1 on 64-bit HP-UX
> : 11.00 systems.  This bug does not exhibit itself on any 32-bit HP-UX
> : 11.00 or HP-UX 10.20 systems that I have built 2.3.0p1 on.  OpenSSH
> : 2.3.0p1 was built with HPs ANSI C compiler with OpenSSL 0.9.6 and zlib
> : 1.1.3.
> :
> : The problem is with the call to vhangup(2) in sshd when interactive
> : sessions are started.  The problem does not occur for non-interactive
> : sessions.  When vhangup(2) is called, the following error is sent to
> : syslog:
> :
> : Mar 13 21:14:37 hpux1100 vmunix: System call 76 (vhangup) was called in
> : a kernel where the
> : Mar 13 21:14:37 hpux1100 vmunix: type of at least one of its arguments
> : is currently
> : Mar 13 21:14:37 hpux1100 vmunix: unspecified.  This is a problem that
> : must be fixed by
> : Mar 13 21:14:37 hpux1100 vmunix: the owner of the system call before the
> : kernel can be
> : Mar 13 21:14:37 hpux1100 vmunix: released.  The process was pid 19386
> : (sshd).
>
> i submitted a defect report on this in october.
>
> but i believe that for STREAMS ptys the vhangup equivalent is considered
> the grantpt(3C) and unlockpt(3C) pair.  can someone confirm this?
>
> i propose the following:
>
> Index: configure.in
> ===================================================================
> RCS file: /var/cvs/openssh/configure.in,v
> retrieving revision 1.265
> diff -u -r1.265 configure.in
> --- configure.in        2001/03/14 00:39:46     1.265
> +++ configure.in        2001/03/14 17:20:09
> @@ -71,7 +71,6 @@
>         AC_DEFINE(DISABLE_SHADOW)
>         AC_DEFINE(IPV4_DEFAULT)
>         AC_DEFINE(IP_TOS_IS_BROKEN)
> -       AC_DEFINE(BROKEN_VHANGUP)
>         AC_DEFINE(NO_X11_UNIX_SOCKETS)
>         no_libsocket=1
>         no_libnsl=1
> Index: defines.h
> ===================================================================
> RCS file: /var/cvs/openssh/defines.h,v
> retrieving revision 1.56
> diff -u -r1.56 defines.h
> --- defines.h   2001/02/24 00:55:05     1.56
> +++ defines.h   2001/03/14 17:20:15
> @@ -406,9 +406,9 @@
>  # endif /* defined(HAVE_XATEXIT) */
>  #endif /* !defined(HAVE_ATEXIT) && defined(HAVE_ON_EXIT) */
>
> -#if defined(HAVE_VHANGUP) && !defined(BROKEN_VHANGUP)
> +#if defined(HAVE_VHANGUP) && !defined(HAVE_DEV_PTMX)
>  #  define USE_VHANGUP
> -#endif /* defined(HAVE_VHANGUP) && !defined(BROKEN_VHANGUP) */
> +#endif /* defined(HAVE_VHANGUP) && !defined(HAVE_DEV_PTMX) */
>
>  #ifndef GETPGRP_VOID
>  # define getpgrp() getpgrp(0)

--
Quentin C. Bracken
UNIX Master Craftsman
Application Development UNIX Services, Sabre Inc.
quentin.bracken at sabre.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.mindrot.org/pipermail/openssh-unix-dev/attachments/20010314/c0adaf3b/attachment.html