[PATCH]: RHL72 --with-skey linking problems

[Pekka Savola]

Hi,

On Red Hat Linux 7.2, it appears one cannot compile with --with-skey; 
configure fails with:

checking for skey_keyinfo... no
configure: error: ** Incomplete or missing s/key libraries.

even though skey 1.1.5 libraries are installed from the location mentioned
in INSTALL.

This has also happened with Solaris it seems:

http://bugzilla.mindrot.org/show_bug.cgi?id=45


A patch to configure.ac is attached here and to bugzilla.  It may require 
some tuning, but the basics seem to be ok and it works here.


Full log below was:
--8<--
configure:5982: checking for skey_keyinfo
configure:6025: gcc -o conftest -g -O2 -Wall -Wpointer-arith 
-Wno-uninitialized   conftest.c -lskey -lutil -lz -lnsl  >&5
/usr/bin/ld: Warning: size of symbol `f' changed from 4 to 26 in 
skeysubr.o
/usr/bin/ld: Warning: type of symbol `f' changed from 1 to 2 in skeysubr.o
collect2: ld returned 1 exit status
configure:6028: $? = 1
configure: failed program was:
#line 5987 "configure"
#include "confdefs.h"
/* System header to define __stub macros and hopefully few prototypes,
    which can conflict with char skey_keyinfo (); below.  */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error.  */
#ifdef __cplusplus
extern "C"
#endif
/* We use char because int might match the return type of a gcc2
   builtin and then its argument prototype would still apply.  */
char skey_keyinfo ();
char (*f) ();

#ifdef F77_DUMMY_MAIN
#  ifdef __cplusplus
     extern "C"
#  endif
   int F77_DUMMY_MAIN() { return 1; }
#endif
int
main ()
{
/* The GNU C library defines this for functions which it implements
    to always fail with ENOSYS.  Some functions are actually named
    something starting with __ and the normal name is an alias.  */
#if defined (__stub_skey_keyinfo) || defined (__stub___skey_keyinfo)
choke me
#else
f = skey_keyinfo;
#endif

  ;
  return 0;
}
configure:6044: result: no
configure:6050: error: ** Incomplete or missing s/key libraries.
--8<--

A _workaround_ is:

--8<--
--- openssh-3.0.2p1.orig/configure	Sun Dec  2 01:38:55 2001
+++ openssh-3.0.2p1/configure	Tue Feb 12 00:09:24 2002
@@ -6153,8 +6153,7 @@
 #endif
 /* We use char because int might match the return type of a gcc2
    builtin and then its argument prototype would still apply.  */
-char skey_keyinfo ();
-char (*f) ();
+char *skey_keyinfo ();
 
 #ifdef F77_DUMMY_MAIN
 #  ifdef __cplusplus
@@ -6171,7 +6170,7 @@
 #if defined (__stub_skey_keyinfo) || defined (__stub___skey_keyinfo)
 choke me
 #else
-f = skey_keyinfo;
+skey_keyinfo;
 #endif
 
   ;
--8<--

(renaming 'f' also would help I guess.)

Tested on 3.0.2p1 but also happens on the latest snapshot.

-- 
Pekka Savola                 "Tell me of difficulties surmounted,
Netcore Oy                   not those you stumble over and fall"
Systems. Networks. Security.  -- Robert Jordan: A Crown of Swords



-------------- next part --------------
--- openssh/configure.ac	Sat Nov  3 21:09:33 2001
+++ openssh.new/configure.ac	Tue Feb 12 01:02:46 2002
@@ -507,9 +507,16 @@
 			LIBS="-lskey $LIBS"
 			SKEY_MSG="yes" 
 	
-			AC_CHECK_FUNC(skey_keyinfo,
-				[],
+			AC_MSG_CHECKING([for s/key support])
+			AC_TRY_RUN(
 				[
+#include <stdio.h>
+#include <skey.h>
+int main() { char *ff = skey_keyinfo(""); ff=""; return 0; }
+				],
+				[AC_MSG_RESULT(yes)],
+				[
+					AC_MSG_RESULT(no)
 					AC_MSG_ERROR([** Incomplete or missing s/key libraries.])
 				])
 		fi