[Bug 3581] New: ssh-keyscan fails with `fdlim_get: bad value` with large file descriptor limit due to type confusion

[bugzilla-daemon at mindrot.org]

https://bugzilla.mindrot.org/show_bug.cgi?id=3581

            Bug ID: 3581
           Summary: ssh-keyscan fails with `fdlim_get: bad value` with
                    large file descriptor limit due to type confusion
           Product: Portable OpenSSH
           Version: 9.3p1
          Hardware: ARM64
                OS: Mac OS X
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: ssh-keyscan
          Assignee: unassigned-bugs at mindrot.org
          Reporter: janerik at fnordig.de

ssh-keyscan fails with an obscure `fdlim_get: bad value` error when the
values returned from `sysconf(_SC_OPEN_MAX)` does not fit into the 32
bits of an `int`.

This happens on my M1 MacBook (with the system-provided `ssh-keyscan`)
where I somehow have an `unlimited` file descriptor limit (which really
is `INT64_MAX` or `9223372036854775807`, note that this seems
non-standard).

Changing the limit to something below `INT_MAX` works.
And then `ssh-keyscan` picks `maxfd=256` anyway.

See this shell session:

```
$ ulimit -n
unlimited
$ ssh-keyscan -t ed25519 github.com
ssh-keyscan: fdlim_get: bad value
$ ulimit -n 4294967295
$ ssh-keyscan -t ed25519 github.com
ssh-keyscan: fdlim_get: bad value
$ ulimit -n 2147483648
$ ssh-keyscan -t ed25519 github.com
ssh-keyscan: fdlim_get: bad value
$ ulimit -n 2147483647
$ ssh-keyscan -t ed25519 github.com
# github.com:22 SSH-2.0-babeld-6732d282
github.com ssh-ed25519
AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl
```

Looking at the code of openssh-portable here[1] shows that `fdlim_get`
returns an `int`, but the fields of `struct rlimit` and the return
value of `sysconf`[2] are `long`, thus 64-bit.

Given that large values like on my machine probably are not set too
often this probably won't affect many people. And a simple workaround
exists.
So maybe this is a wontfix, but I figured I'd still report it and let
you decide.

[1]:
https://github.com/openssh/openssh-portable/blob/b4ac435b4e67f8eb5932d8f59eb5b3cf7dc38df0/ssh-keyscan.c#L129-L130
[2]: `SSH_SYSFDMAX` is defined as `sysconf(_SC_OPEN_MAX)`

-- 
You are receiving this mail because:
You are watching the assignee of the bug.