Apple's SSH x OpenSSH (brew) x CTK x Security Key types

[Jan Schermer]

Hi,
What I was trying to do (apart from toying with stuff) was to get a realiable, single, portable/importable credential that would be universally available whenever I need it but in normal operation would be either stored in or wrapped by Secure Enclave (this means EC keys), instead of provisioning 5 resident FIDO keys, one Secretive SE-wrapper key and a backup key. (I know, I could use certificates, and maybe I will!).
Why don’t I just use the brew version with pkcs11/ykcs11?
a) Because it adds all the keys in the PIV token including deleted keys that are not listed anywhere (I only got rid of those by resetting the applet completely) and the attestation key
 * thankfully the attestation key refuses to sign arbitrary data, as it should
b) Because I’d rather it be bound to the Secure Enclave and only have a token as a backup or for stronger security requirements

This got me into a rabbit hole :-)

Sorry in advance if this is not 100% suitable for this mailing list, but I’m at loss where to inquire about this. It looks like the best course of action is to just wait a bit for things to mature, so feel free to ingest it as just a curiosity.

Also there are several parts/questions to this story:
———

1) macOS CTK support for “virtual smartcards"
I noticed that some functionality was added in recent macOS versions, possibly even only to the recent betas.
In particular, the ability to use the CryptoTokenKit backend to store or generate credentials, including wrapping the keys using Secure Enclave i.e.:

sc_auth create-ctk-identity -l SSH -k p-256-ne -t bio -N username -E username at example.com <mailto:username at example.com>
or (more interestingly)
sc_auth import-ctk-identities -f credentials.pfx -t bio

This is very nice, I currently use Secretive.app to do this but this makes backups/portable keys possible and is what triggered this whole endeavour.
(WARNING: importing a certificate that is capable of logging you in via sc_auth most likely results in inability to authenticate on lockscreen other than using TouchID as it is always “plugged in", requires a bogus PIN and the interface doesn’t allow you to type in password anymore!)

I think the purpose for this is to better support credential enrollment from SSO/MDM etc. (or are they planning to completely kill keychain?), but that’s just my theory.
———

2) ssh-keychain.dylib now exposes EC keys in CTK as Security Keys (w00t?)
quoting from man:
"By default, all valid (RSA for PKCS#11 and ecdsa256 for Secure Key module) identities from all SmartCards and persistent tokens currently available in the system are provided."

export SSH_SK_PROVIDER=/usr/lib/ssh-keychain.dylib
ssh-keygen -K
ssh -i ecdsa_sk_rk user at example.com <mailto:user at example.com>
and you get logged in with an ECDSA key in the PIV applet (but see point 4 below)

What Apple has implemented here is pretty... weird.
They implemented SK emulation (that doesn’t support RSA or generating keys or ECP-384) but left the PKCS#11 interface RSA-only ¯\_(ಠ_ಠ)_/¯ 
Am I the only one completely baffled by this approach?
If I were an optimist, I’d guess that they are going to support passkeys for SSH at some point and this is a step towards that?

This should supposedly (maybe primarily) work for other CTK-backed identities, like PIV tokens, but I couldn’t get this to work. Or rather it seems to work with no error, but doesn’t _actually_ work:

This is with Yubikey 5C and certificate in 9a slot (9c does the same)
debug1: Server accepts key: id_ecdsa_sk_rk ECDSA-SK SHA256:0dttd879INvMlZ92xl4NOIkJ2AJUksEAsup0UgSqu5k explicit authenticator
debug3: sign_and_send_pubkey: using publickey-hostbound-v00 at openssh.com with ECDSA-SK SHA256:0dttd879INvMlZ92xl4NOIkJ2AJUksEAsup0UgSqu5k
debug3: sign_and_send_pubkey: signing using sk-ecdsa-sha2-nistp256 at openssh.com SHA256:0dttd879INvMlZ92xl4NOIkJ2AJUksEAsup0UgSqu5k
Confirm user presence for key ECDSA-SK SHA256:0dttd879INvMlZ92xl4NOIkJ2AJUksEAsup0UgSqu5k
debug3: start_helper: started pid=11453
debug3: ssh_msg_send: type 5
debug3: ssh_msg_recv entering
debug1: start_helper: starting /usr/libexec/ssh-sk-helper
debug1: process_sign: ready to sign with key ECDSA-SK, provider /usr/lib/ssh-keychain.dylib: msg len 363, compat 0x4000000
debug1: sshsk_sign: provider "/usr/lib/ssh-keychain.dylib", key ECDSA-SK, flags 0x21
debug1: sshsk_open: provider /usr/lib/ssh-keychain.dylib implements version 0x000a0000
debug1: main: reply len 63
debug3: ssh_msg_send: type 5
debug3: reap_helper: pid=11453
User presence confirmed
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.

This is with the credential imported into CTK (using the same downloaded private resident key):

debug1: Server accepts key: id_ecdsa_sk_rk ECDSA-SK SHA256:0dttd879INvMlZ92xl4NOIkJ2AJUksEAsup0UgSqu5k explicit authenticator
debug3: sign_and_send_pubkey: using publickey-hostbound-v00 at openssh.com with ECDSA-SK SHA256:0dttd879INvMlZ92xl4NOIkJ2AJUksEAsup0UgSqu5k
debug3: sign_and_send_pubkey: signing using sk-ecdsa-sha2-nistp256 at openssh.com SHA256:0dttd879INvMlZ92xl4NOIkJ2AJUksEAsup0UgSqu5k
Confirm user presence for key ECDSA-SK SHA256:0dttd879INvMlZ92xl4NOIkJ2AJUksEAsup0UgSqu5k
debug3: start_helper: started pid=12187
debug3: ssh_msg_send: type 5
debug3: ssh_msg_recv entering
debug1: start_helper: starting /usr/libexec/ssh-sk-helper
debug1: process_sign: ready to sign with key ECDSA-SK, provider /usr/lib/ssh-keychain.dylib: msg len 363, compat 0x4000000
debug1: sshsk_sign: provider "/usr/lib/ssh-keychain.dylib", key ECDSA-SK, flags 0x21
debug1: sshsk_open: provider /usr/lib/ssh-keychain.dylib implements version 0x000a0000
   (At this point, TouchID popups requires confirmation) 
debug1: main: reply len 129
debug3: ssh_msg_send: type 5
debug3: reap_helper: pid=12187
User presence confirmed
debug3: send packet: type 50
debug3: receive packet: type 52
Authenticated to example.com <http://example.com/> ([1.2.3.4]:22) using “publickey”.
———

3) Difference in SSH_SK_PROVIDER handling between Apple’s SSH and OpenSSH via brew

In particular what I demonstrated above:
SSH_SK_PROVIDER=/usr/lib/ssh-keychain.dylib
ssh -i ecdsa_sk_rk user at example.com <mailto:user at example.com>

does not work with homebrew:
debug1: process_sign: ready to sign with key ECDSA-SK, provider internal: msg len 363, compat 0x4000000
debug1: sshsk_sign: provider "internal", key ECDSA-SK, flags 0x21
debug1: sk_probe: 0 device(s) detected

It looks like SSH_SK_PROVIDER is not passed to the ssh-sk-helper or it doesn’t use it in brew version? (Is there some sort of whitelist of allowed library paths like for ssh-agent?)
I didn’t actually bother with passing it to global environment (I expect it to both work without it and make no difference) or trying ssh-agent, I got tangled in it enough for today.
UPDATE:
using ssh -oSecurityKeyProvider=/usr/lib/ssh-keychain.dylib makes it work with brew version, it’s just the environment variable that doesn’t work.
———

4) Compatibility of the SK and non-SK keys?

Secretive.app makes PIV credentials available in the agent, the key I am testing with looks like this (and of course looks the same when using PKCS#11):
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGaoNBTGBrSRYbRjPMVw9bBJtqmrw/WNWTX3MfjmTM8Ugj4alqffbqfUsTmpiR42AjTpyroTSdRCt4DvnO0dYxY= Key-For-PIV-Authentication-(user)@secretive.Hostname.local

But what ssh-keygen -K produces is the -sk variant:
sk-ecdsa-sha2-nistp256 at openssh.com AAAAInNrLWVjZHNhLXNoYTItbmlzdHAyNTZAb3BlbnNzaC5jb20AAAAIbmlzdHAyNTYAAABBBGaoNBTGBrSRYbRjPMVw9bBJtqmrw/WNWTX3MfjmTM8Ugj4alqffbqfUsTmpiR42AjTpyroTSdRCt4DvnO0dYxYAAAAEc3NoOg== ssh:

"AAAAIbmlzdHAyNTYAAABBBGaoNBTGBrSRYbRjPMVw9bBJtqmrw/WNWTX3MfjmTM8Ugj4alqffbqfUsTmpiR42AjTpyroTSdRCt4DvnO0dYxY” matches both, I suppose that’s the actual key material?
So are those cryptographically the same*? Can I just paste a “header" and “footer" to any key and call it sk-*?
(Related question - the downloaded resident private key has slight differences between every download from the token, does that sound right?)
Would it be possible and useful to make a public key type accepting both somehow? Or maybe the right question is - can the non-sk authorized key be made to work with a security key? I believe it’s still the same crypto* and this would bring compatiblity to OpenSSH versions that don’t support sk-*. The client simply needs to present both variants of the pubkey.

  * More like a proven fact. I can reimport this EC key, use a different Yubikey with it or import it into CTK and it still works the same, so there’s nothing not already contained in the private key that’s needed. Separating those two public key types in the first place thus seems unnecessary and artificially incompatible, I wonder what the rationale is?

This should also be a clear warning sign for anyone only depending on “sk-*” key type instead of checking attestation when onboarding.
———

5) Any development planned to support passkeys?
Using passkeys would just solve most of my problems at this point, but I’m not aware of a provider library making them available (there’s a singular bug/request somewhere in yubico’s libfido for it that’s several years old, I realize that’s a more suitable place and I’ll file an Issue for it).
There’s support in Blink, but it’s a paid feature and I wasn’t able to test it (nor do I want to use Blink).
———


I hope all this was at least interesting, any comments are appreciated

Jan