Empty (zero byte) SSH host keys

[Benjamin Drung]

Am Donnerstag, den 07.01.2016, 03:18 -0500 schrieb Nico Kadel-Garcia:
> On Wed, Jan 6, 2016 at 1:08 PM, Benjamin Drung
> <benjamin.drung at profitbricks.com> wrote:
> > Hi,
> > 
> > We create virtual machine image templates by doing automated
> > minimal
> > installations of different Linux distributions (via
> > preseed/kickstarter/autoyast). At the end of the installation, we
> > remove the SSH host keys (rm -f /etc/ssh/ssh*_key*). Fresh SSH host
> > keys will be generated on the first boot of the image instances.
> > This
> > is done by adding a "dpkg-reconfigure openssh-server" call in
> > /etc/rc.local (which calls ssh-keygen) on Debian/Ubuntu and by the
> > init
> > script of sshd on the other distributions.
> > 
> > This leads to working SSH server running on the virtual machines
> > most
> > of the times, but sometimes the SSH connection fails with
> > "connection
> > reset by peer". The investigation of Debian 7 "wheezy" images
> > showed
> > that these faulty machines have empty (zero byte) SSH host key
> > files.
> > These files do not exist before the machines are started, but they
> > do
> > exist before "dpkg-reconfigure openssh-server" is called.
> > 
> > So it seems that some process creates these empty SSH host key
> > files.
> > Can you help to further debugging this strange behavior? Does sshd
> > create SSH host keys?
> 
> It could be a filesystem issue in your image generation process. Are
> you doing a "sync" before recording your image? And are you using one
> of the more fragile, "optimized" filesystems like ReiserFS, which is
> infamous for zeroing files?

All images use ext4 as filesystem.

The image creation works as following:
1) Create a raw HDD image (2 GiB in size)
2) Launch qemu to do a preseed/kickstarter/autoyast installation
2b) as part of the installation remove the SSH host keys (via a
late_command or post command)
2c) wait for the installation to finish / exit of qemu
3) mount the raw image
4) copy the installer logs and remove them
5) unmout the raw image
6) run zerofree on the partition
7) convert the raw image to qcow2 and upload/distribute it

When an instance is created from this template, following steps are
done:

1) create a block device (could be larger than the image)
2) copy the template to the block device
3) resize the root partition and add a tmpfs partition
4) mount the resized partition
5) (optionally) inject a authorized_key file
6) set the root password
7) verify that no SSH host keys are on this partition
8) unmount the partition
9) launch qemu

> Until you discover the issue, you *could* edit the OpenSSH init
> script, or /etc/sysconfig/sshd, to clear the hostkeys if they're
> empty
> and before new keys would be auto-generated.

That's exactly the workaround that we are currently testing.

-- 
Benjamin Drung
System Developer
Debian & Ubuntu Developer

ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin

Email: benjamin.drung at profitbricks.com
URL:  http://www.profitbricks.com

Sitz der Gesellschaft: Berlin.
Registergericht: Amtsgericht Charlottenburg, HRB 125506B.
Geschäftsführer: Andreas Gauger, Achim Weiss.