OpenSSH w/ kth-krb4 on AIX

[Richard Levitte - VMS Whacker]

From: sperber at informatik.uni-tuebingen.de (Michael Sperber [Mr.  Preprocessor])

sperber> >>>>> "Moiself" == Michael Sperber <sperber at informatik.uni-tuebingen.de> writes:
sperber> 
sperber> Moiself> Hi,
sperber> 
sperber> Moiself> I'm suffering from a memory corruption problem when
sperber> Moiself> compiling OpenSSH 3.0.2p1 with kth-krb4 1.1 on AIX
sperber> Moiself> 4.3.2 and 4.3.3.  The symptom is that the file name
[...]
sperber> Here's the source of the problem: krb4 and openssl both define
sperber> RC4_INT, and they define it differently.  OpenSSH happens to call some
sperber> routines from one, and some from the other.  Poing!  I don't know what
sperber> to do about this in general.  Suggestions?

It's been suggested a few days ago that recent versions of kth-krb4
can be configured to use OpenSSL to get the crypto routines.  Doing so
is very likely to resolve the problem.

I wonder, where exactly does lib/des/rc4* come from?  SSLeay?  I just
checked, and when looking at the Configure from OpenSSL (latest CVS
update) and from SSLeay 0.8.1b (which is basically the absolutely
earliest SSLeay we have in the OpenSSL repository), the aix targets
have exactly the same configuration options (BN_LLONG and RC4_CHAR).
Back in SSLeay times, one could find the following in rc4.org (and the
distributed rc4.h, which was rebuilt from rc4.org by the configuration
script):

    #define RC4_INT unsigned int

And as long as rc4.h got rewritten properly for the different
architectures, all was fine.  However (and this is very unfortunate,
even if it was a nice thought), SSLeay was designed in such a way that
crypto algorithm implementations could be picked out and form
independent libraries, of which libdes is probably the most well-known
(for the picky: it was actually the other way around, libdes and the
like were basically grafted into SSLeay).  However, since those lack
the SSLeay/OpenSSL configuration script, you would end up having
differences on some platforms between lib{crypto} and the same
routines in libcrypto.  And this only becomes visible when the two are
suddenly combined, something I don't think has happened before OpenSSH
started using both OpenSSL's libcrypto and kth-krb4 (which in this
case was NOT built against OpenSSL's libcrypto).

All this is quite unfortunate, and we (the OpenSSL team) have battled
with the old des routines (which we unfortunately messed up for a bit
of time) and have finally come up with a method to avoid all clashes
that we can think of.  Do we need to do that for all other crypto
implementations that were present in SSLeay, or should kth-krb4 move
to require the presence of OpenSSL?  I personally would prefer the
latter (less work for us, that's why :-)), but I'm up for listening.

-- 
Richard Levitte   \ Spannvägen 38, II \ LeViMS at stacken.kth.se
Redakteur at Stacken  \ S-168 35  BROMMA  \ T: +46-8-26 52 47
                    \      SWEDEN       \ or +46-733-72 88 11
Procurator Odiosus Ex Infernis                -- poei at bofh.se
Member of the OpenSSL development team: http://www.openssl.org/
Software Engineer, GemPlus:             http://www.gemplus.com/

Unsolicited commercial email is subject to an archival fee of $400.
See <http://www.stacken.kth.se/~levitte/mail/> for more info.