[Bug 3530] New: AC_CHECK_FUNCS of libfido2 functions generate wrong results if not using pkg-config

[bugzilla-daemon at mindrot.org]

https://bugzilla.mindrot.org/show_bug.cgi?id=3530

            Bug ID: 3530
           Summary: AC_CHECK_FUNCS of libfido2 functions generate wrong
                    results if not using pkg-config
           Product: Portable OpenSSH
           Version: 9.1p1
          Hardware: All
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: Build system
          Assignee: unassigned-bugs at mindrot.org
          Reporter: yzhangad at gmail.com

When autoconf uses `AC_CHECK_FUNCS` to test the existence of necessary
libfido2 functions, it appends `$LIBFIDO2` to `$LIBS`:
https://github.com/openssh/openssh-portable/blob/a20b7e999773e6333c8aa9b0a7fa41966e63b037/configure.ac#L3248.
In an environment without `pkg-config`, this makes `-lfido2 -lcbor`
appear at the end of the linker flags.

The problem of this: `libcbor` is not the only dependency of
`libfido2`: it also depends on, e.g. `libcrypto`, which is part of
OpenSSL, and that can be configured via the `--with-ssl-dir` option.
When OpenSSL and `libfido2` are built statically, the existing
`configure.ac` makes `-lcrypto` go before `-lfido2`, making the linkage
break, thus causing false negative results of `AC_CHECK_FUNCS`.

The fix should consist just of replacing `LIBS="$LIBS $LIBFIDO2"` with
`LIBS="$LIBFIDO2 $LIBS"`.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.