Host key verification (known_hosts) with ProxyJump/ProxyCommand

[Stuart Longland VK4MSL]

Hi all,

I noticed a bit of an odd issue with maintaining `known_hosts` when the 
target machine is behind a bastion using `ProxyJump` or `ProxyCommand` 
with host key clashes.

Client for me right now is OpenSSH_9.3p1 on Gentoo Linux/AMD64.  I'm a 
member of a team, and most of us use Ubuntu (yes, I'm a rebel).  Another 
team who actually maintain this fleet often access the same machines via 
Windows 10/11 boxes (not sure if they use native OpenSSH or WSL).  I 
rather suspect this issue actually is not platform-specific.

Target machines are using OpenSSH on Debian/ARMHF (the exact version 
varies with the exact OS version) -- hardware is essentially 
industrialised Raspberry Pis.

The bastions are typically OpenWRT-based (Teltonica) routers with 
Dropbear SSHd.

We share a configuration tree via a git repository which contains `Host` 
entries for each of the target machines and the intermediate bastion hosts.

The target machines are mostly using "private" address space in the 
172.16.0.0/12 subnet (although some are using 172.40.0.0/16 addresses, 
because some goose thought all 172.0.0.0/8 subnets were "private").  The 
bastion hosts run `dhcpd` and in many cases, the target acquires its 
address via DHCP (yes, super bad idea).

That means that in many cases, multiple _different_ OpenSSH servers, 
have the "same" IPv4 local address.  Since when using `ProxyJump`, this 
address is recorded along side the host's public key in the user's 
`known_hosts` file, you can imagine when one logs into one server via 
one bastion, then tries to log in to a different server via a different 
bastion if that second server has the same local IPv4 address.

The crux of this is that we cannot assume the local IPv4 address is 
unique, since it's not (and in many cases, not even static).

In the case of `ProxyCommand`, the IPv4 address of the target may not 
even be obvious to the SSH client process.

-- Possible solutions / work-arounds using existing OpenSSH client --

I looked around for a solution, I ruled out turning off 
`StrictHostKeyChecking` (as seen on ServerFault) as a terrible idea 
asking for a Man-In-The-Middle attack.

DNS might "solve" the problem, but is likely to be messy to implement 
(bastion hosts are resource constrained, not sure if they do dynamic DNS 
for their LAN clients).

I know I'll get push-back from the other team if I try to mandate unique 
local IPv4 addresses.  (This is the same team that unwittingly decided 
to rely on DHCP static assignment to "do the right thing".)

Link-local IPv6 is a tempting prospect: I have used this to get into a 
target node when its DHCP client has gone AWOL leaving IPv4 
unconfigured, and being derived from the MAC address, *should* be 
globally unique, but this assumes a dual-stack LAN.  (And the 
engineering team who look after these are likely to baulk at this.  They 
barely understand IPv4!)

Port forwarding will require a lot of manual piss-farting around on the 
router's config webpage… and will likely break if the embedded DHCPd 
decides to not assign the static IP the target machine was supposed to get.

In the `ssh_config` man page, I see there is a `KnownHostsCommand` 
option, which could possibly be employed here, however since the files 
are "shared" by multiple users, there's the issue of paths, since I'll 
bet the `KnownHostsCommand` is relative to ${PWD} and not ~/.ssh/config 
or any config file imported by it.

User or Global `known_hosts` won't work due to the format of the file 
used (it assumes a unique endpoint IP address, which we know is not 
unique).  (I have `HashKnownHosts` turned off on this Gentoo machine, my 
workplace laptop has it turned on due to Ubuntu's default.  Not sure if 
this hash takes into account `ProxyJump` paths or `ProxyCommand` options.)

-- Possible solutions that require OpenSSH client changes --

One way that might work would be to embed the effective 
`ProxyJump`/`ProxyCommand` path in the "host name" stored in 
`known_hosts` -- will look ugly as sin, but at least the client can 
"uniquely" identify each server, and determine which key to use for 
validation.

e.g. you might have in known_hosts
172.16.1.2{ProxyJump user at 10.20.30.40,user2 at 192.168.123.45} <algo> <key>
172.16.1.3{ProxyCommand user at 10.20.30.40:nc 192.168.234.56 22} <algo> <key>
the {} part encodes the path by which you reach the host.

Alternative might be an "ExpectHostKey" option that can be put in 
~/.ssh/config or specified with "-o ExpectHostKey=…" that tells the SSH 
client "ignore your known_hosts file, the host *will* be using this 
key".  So if you know the public key (e.g. you did a `ssh_keyscan`), you 
can either:

put in .ssh/config:

Host mytarget
	Hostname 172.16.1.2
	ProxyJump user2 at bastion2
	ExpectHostKey ecdsa-sha2-nistp256 AAAA…=

Host bastion2
	Hostname 192.168.123.45
	ProxyJump user at bastion1
	ExpectHostKey ecdsa-sha2-nistp256 AAAA…=

Host bastion1
	Hostname 10.20.30.40
	ExpectHostKey ecdsa-sha2-nistp256 AAAA…=

OR, you might specify it on the command line (assuming the bastions are 
"known")

ssh -o ExpectHostKey="ecdsa-sha2-nistp256 AAAA…=" \
	-J user at bastion user at target

Bonus with this latter approach is that in a config sharing environment 
using a SCM (whether it be git, Subversion, CVS… whatever), assuming 
that repository was protected and "trusted", it would enable all members 
of a team to automatically "trust" the host key with minimal 
infrastructure set-up.

Are any of the above ideas feasible?  Did I miss an obvious solution to 
this?
-- 
Stuart Longland (aka Redhatter, VK4MSL)

I haven't lost my mind...
   ...it's backed up on a tape somewhere.