[PATCH] Enabling ECDSA in PKCS#11 support for ssh-agent

[Mathias Brossard]

Hi,

I have made a patch for enabling the use of ECDSA keys in the PKCS#11
support of ssh-agent which will be of interest to other users.

I have tested it with P-256 keys. P-384 and P-521 should work
out-of-the box. The code is ready for non-FIPS curves (named or
explicit), but OpenSSH currently limits ECDSA to those 3 curves.

At high level it works like the support for RSA, but because of
differences in OpenSSL between RSA and EC_KEY, implementation has a
few differences. The RSA and RSA_METHOD structures are exposed and the
existing ssh-pkcs11 code uses that to create an RSA_METHOD object for
each key.

Because of APIs (in addition to ECDSA support) needed by the patch
this currently works with:

- LibreSSL >= 2.2.2: until LibreSSL 2.1.2 (which is the what I am
  testing for), the ECDSA_METHOD structure was defined in a private
  header. But the LIBRESSL_VERSION_NUMBER constant was not updated
  until 2.2.2.

- OpenSSL >= 1.0.2: creating your own ECDSA_METHOD is not possible
  before because the ECDSA_METHOD structure if opacified. In OpenSSL
  1.0.2, they added the option to create new ECDSA_METHOD object if
  this is detectable with the ECDSA_F_ECDSA_METHOD_NEW define.

A few notes to understand the patch:

- A few places assumed RSA keys, I added a key type field and use it
  to handle the differences. I also renamed some function to reflect
  their link to RSA.

- I moved some code out of pkcs11_rsa_private_encrypt into a separate
  function pkcs11_login to share it with pkcs11_ecdsa_sign

- For EC_KEY, the pointer to the struct pkcs11_key object is not in
  the method but in the EC_KEY itself using ECDSA_set_ex_data and
  ECDSA_set_ex_data. This allows having a single ECDSA_METHOD for
  all keys.

- Unlike the RSA_METHOD, ECDSA_METHOD does not include a "finish"
  method to clean up the associated data. This was only a problem for
  ssh-pkcs11-helper.c that called key_free on struct sshkey objects
  created by ssh-pkcs11.c. To work around that I added a function
  pkcs11_del_key(struct sshkey *) to the list of functions exported by
  ssh-pkcs11.c that allows us to properly clean up ECDSA keys.

I tried to:
- be as consistent as possible with the RSA part,
- minimize the size of the patch and the number of locations,
- document some of the additional quirks specific to ECDSA.

I added this patch and text as
https://bugzilla.mindrot.org/show_bug.cgi?id=2474

Sincerely,
-- 
Mathias Brossard