OpenSSH on HP-UX 11 with TCB

[Ged Lodder]

Hi,

an updated and more civilized post (to my one and only previous one) on
getting OpenSSH to work on HP-UX 11 using the TCB. I used the HP ANSI C
compiler. 


Firstly, I needed to download, compile and install OpenSSL, EGD and
ZLib.

Specific issues:
	
	configure did not handle hpux 11
	login.c did not compile
	makefile did not use $(CFLAGS)
	sshd did not compile (pam issues), I wanted to use supplied pam library
	I use long passwords, > 8 chars, needed bigcrypt()


Solutions for configure:

I added the following to the configure script:

	
*-*-hpux11*)
        if test -z "$GCC"; then
                CFLAGS="$CFLAGS -Ae"
        fi
        CFLAGS="$CFLAGS -D_HPUX_SOURCE"
        cat >> confdefs.h <<\EOF
#define IPADDR_IN_DISPLAY 1
EOF

        cat >> confdefs.h <<\EOF
#define USE_UTMPX 1
EOF

        echo $ac_n "checking for HPUX trusted system password
database""... $ac_c" 1>&6
echo "configure:1301: checking for HPUX trusted system password
database" >&5
        if test -f /tcb/files/auth/system/default; then
                echo "$ac_t""yes" 1>&6
                cat >> confdefs.h <<\EOF
#define HAVE_HPUX_TRUSTED_SYSTEM_PW 1
EOF

                LIBS="$LIBS -lsec"
                echo "configure: warning: This configuration is
untested" 1>&2
        else
                echo "$ac_t""no" 1>&6
                cat >> confdefs.h <<\EOF
#define DISABLE_SHADOW 1
EOF

        fi
        MANTYPE='$(CATMAN)'
        mansubdir=cat
        ;;


Solution for login.c

Missing an opening brace at line 213, added it. Code excerpt below:

#if defined(HAVE_ADDR_IN_UTMPX)
        if (addr) { 				/* Added a brace here - Ged */
                switch (addr->sa_family) {


Solution for Makefile:
Changed Makefile.in and added $(CFLAGS) to the compile lines for all the
executables.

Solution for sshd:

HP-UX does not have a pam_getenvlist call. Changed the
fetch_pam_environment function to
return NULL if _HPUX_SOURCE was defined.

/* Return list of PAM environment strings */
char **fetch_pam_environment(void)
{
#ifndef _HPUX_SOURCE    /* HP-UX has not implemented  this */
    return(pam_getenvlist((pam_handle_t *)pamh));
#else
    return(NULL);
#endif
}


Solution for long passwords:

Need a couple of includes, namely <hpsecurity.h> and <prot.h>
Added the following to defines.h

#ifdef HAVE_HPUX_TRUSTED_SYSTEM_PW
# include <hpsecurity.h>
# include <prot.h>
#endif

Modified auth-passwd.c (not the most elegant fix, assumed HP will not
use MD5 passwds if
trusted). Code below:

#ifdef HAVE_MD5_PASSWORDS
    if (is_md5_salt(salt))
        encrypted_password = md5_crypt(password, salt);
    else
        encrypted_password = crypt(password, salt);
#else /* HAVE_MD5_PASSWORDS */
# ifdef HAVE_HPUX_TRUSTED_SYSTEM_PW
    encrypted_password = bigcrypt(password, salt);
# else
    encrypted_password = crypt(password, salt);
#endif /* HAVE_HPUX_TRUSTED_SYSTEM_PW */
#endif /* HAVE_MD5_PASSWORDS */


Still some issues with man pages and warnings during compilation.
Get syslog error=> error: ioctl I_PUSH ttcompat: Invalid argument
Get syslog error=> Cannot delete credentials: Permission denied

I had to have root permissions to run configure successfully as /tcb on
my system has permissions 500

Run configure with (at least) --with-rsh=/usr/bin/remsh


Thanks 

Ged Lodder -- lodder at yacc.com.au --