array lengths not known at compile time

[Darren Tucker]

On Mon, 25 Jan 2021 at 08:00, Tim Rice <tim at multitalents.net> wrote:
> I've been building OpenSSH on UnixWare 7 using the native compiler
> for a long time. I pulled the latest source yesterday and got this error.
[...]
> UX:acomp: ERROR: "/opt/src/networking/openssh/openssh/sntrup761.c", line 298: integral constant expression expected
> In the source we see lines like this
> uint16 R2[(len+1)/2];
[...]
> UnixWare's USL compiler doesn't know what to do when "len" is not known
> at compile time.

OpenSSH has historically been ANSI C / C89, and we've largely avoided
relying on anything outside of that.  (There are some exceptions such
as the XMSS post-quantum key exchange, but that's experimental and not
enabled by default).

Variable length arrays like this are not part of C89.  They seem to be
supported as an extension by more or less every GCC even in c89 mode
(I tested 3.0.4), so this will only be an issue for non-gcc compilers.

> I know how to "fix" this but as this is 2021 and UnixWare 7D2M1
> has a GCC 7.3.0 available, I'm not sure it is worth the effort.
> It may be time to drop support for old crufty compilers.
>
> The question in my mind is, are UnixWare (and Openserver 6) the only
> platforms that will be tripped up on this?

My guess is that this won't be the only one.  Fixing it has a
complicating factor for sntrup761.c in particular as it's generated by
a script from the original code from SuperCop, so any non-trivial
edits will be a significant maintenance headache.

> But what about IRIX, HP-UX, SGI, Tru64?
> I'm guessing they updated their compilers long ago but I do not know.

The (very old) HP ANSI C compiler on my (very old) C-class workstation
accepts it.  Can't speak to any of the others.

I like the fact that now we have --without-openssl and --without-zlib,
you can build a functional OpenSSH (albeit with a limited set of
supported options) using only a C89 compiler and a half-decent make.
I would like to continue supporting that, as long as doing so does not
compromise supporting modern platforms.

To that end, I think we should disable sntrup761 if the compiler
doesn't support variable length arrays.  We can do that by providing
some no-op KEX functions that just return an error (similar to what we
do in kexecdh.c) and a couple of ifdefs so I don't think it'll be too
much of a headache.

-- 
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860  37F4 9357 ECEF 11EA A6FA (new)
    Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.