removing keys from ssh-agent without having key file

[Matthew Boedicker]

ssh-agent does not allow you to remove individual keys without having the
key file that was added. To remove these keys the user must remove all keys
with ssh-add -D.

Would a patch to make ssh-add skip the existence check for the file be
considered?

The specific use case is that a USB drive is mounted with the key, the key
is added to the agent then the USB drive is unmounted.