Howto log multiple sftpd instances with their chroot shared via NFS

[Peter Stuge]

Hi Hildegard,

Hildegard Meier wrote:
> > Does the patch idea seem viable?
..
> I have no capacity to follow the OpenSSH security issues myself an
> then if needed re-compile newer patched versions (and not even then
> apply your patch additionally every time to it :)

Fair enough, although it's not too much effort..

OpenSSH has a few releases per year (including security updates), there's
a release announcement mailing list with no other traffic, and the build
commands I sent need only minimal adjustment for new version numbers in
the git checkout command - or remove the git checkout command to always
build the very latest code, then the commands stay identical, including
applying the patch. The patch is small so should apply cleanly for some
time.


> We do here industry production service and need to stick with
> vanilla distribution OpenSSH and leave the delivery security
> patches to Ubuntu.

Ack. I consider upstream releases to have higher quality, because
distributions usually ship OpenSSH with their own patches, which
every now and then introduce some problems.

And Ubuntu like all other distributions usually take maybe a day or two
even for critical updates, full new releases can take months or years,
so a local build on release day is much faster, but you are right:

A local build does require tracking releases and running commands other
than the standard update process, and while few packages are as easy to
build as OpenSSH I understand that it can still be out of scope.



> > BindsTo=sshd.service
> > mkdir /etc/systemd/system/sshd.service.wants
> > ln -s ../sftpd.service /etc/systemd/system/sshd.service.wants/
> > Keep After=sshd.service in sftpd.service.
> 
> Thanks for the hint, I will look into it, I am not so experienced
> with systemd yet.

The systemd dependency data model is simple but very capable and allows
for neat yet precise configuration. I like it a lot. Can recommend.


Hildegard Meier wrote:
> My workaround is the following:
..
> mount --bind /var/data/dev/<username>     /var/data/chroot/<username>/dev
> 
> so /var/data/chroot/<username>/dev is now effectively local on the
> sftp server, not anymore on nfs mount.

This solution is perfect if per-user mounts are acceptable.


> Then change the syslog-ng config
> 
> from
> source s_chroot_<username>     { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
> 
> to
> source s_chroot_<username>     { unix-stream("/var/data/dev/<username>/log" optional(yes) ); };
> 
> (this is not strictly needed, but I think it's nice having syslog-ng
> definitely now only reading from local file, guaranteed not from nfs
> mount anymore)

I agree, I think this is an important step for reliability.


> I think I will create systemd unit files for every bind mount
> (I experienced systemd dependency cycles with bind mounts in /etc/fstab
> on Ubuntu18 boot before anyway, so systemd units are needed anyway).

If users have local accounts then a generator[1] can create units
dynamically (on boot and on systemctl daemon-reload), otherwise units
must be created and deleted explicitly by user account management.


I've attached a suggested generator script that reads /etc/passwd and
creates units for all members of the group set at the top of the script.
The sshd_config you sent matches on group sftp-cust so I used that.

The generator can be tested in a temporary directory:

dir=$(mktemp -d); ./sftplog-generator.sh "$dir" && ls -lR "$dir"


When all is good copy the script to /etc/systemd/system-generators/
and after running  systemctl daemon-reload  the generated units will
be in /run/systemd/system-generators/ (never stored on disk) and have
all been loaded.

systemctl restart sftpd.service  then mounts everything before
starting the new sftpd process.


> Maybe one can define a systemd dependency so that when you unmount
> the nfs mount /var/data/chroot/, all the bind mounts under it are
> automatically unmounted before (sftpd needs to be stopped first of
> course, otherwise mount would be busy),

Yes, but it requires using  systemctl stop var-data-chroot.mount
to unmount. systemd doesn't know about unsuccessful attempts to
directly umount a busy /var/data/chroot, but when stopping the
mount unit it knows to first stop the dependency tree. sftpd can
be included there.


> and also the bind-mounts automatically be mounted after /var/dat/chroot
> is mounted again.

Yes, that's a great idea. So looking at the dependencies:

systemd implicitly[2] adds a Requires= dependency with After= ordering
for directories above the target of mount units, so the generated units
get this for /var/data/chroot automatically, ensuring that /var/data/chroot
is mounted first.

The generator creates units with Before=sftpd.service ordering, it creates
Wants= dependencies for var-data-chroot.mount on each bind mount using a
symlink in var-data-chroot.mount.wants/, and finally it creates
BindsTo= dependencies for sftpd.service on each bind mount using so-called
drop-in[3] files in sftpd.service.d/.

The Wants= symlinks automatically mount the bind mounts when /var/data/chroot
is mounted, and the BindsTo= drop-in files ensure that when sftpd.service
is started all bind mounts are mounted first, and also that if any bind
mount is unmounted then sftpd.service is stopped.

A good example of the dependency model capabilities!


> I think I will implement a nagios monitoring check that checks if
> for every username /var/data/chroot/<username>/dev is a mountpoint,
> so logging is assured.

Can do, but with the above dependencies systemd will never start sftpd
without all bind mounts and will stop sftpd if any bind mount is unmounted.

If stopping sftpd.service is too violent then change BindsTo=${unit}
to Requires=${unit} near the bottom of the generator, but add
BindsTo=var-data-chroot.mount to sftpd.service, otherwise sftpd will not
be stopped when stopping var-data-chroot.mount, causing that to fail.

I like how BindsTo= ensures that sftpd either works fully or not at all,
but that may not suit everyone.



> Possible alternative
> --
> I guess (not tested) an alternative could be to hardlink (so needs
> to be on the same (nfs) file system) every
> /var/data/chroot/<username>/dev/log

Unfortunately it doesn't work; the hardlinks must be (re)created after
syslog-ng creates the general socket, but syslog-ng A and syslog-ng B
will create different general socket inodes, the later will overwrite
the earlier, so it's back to square one.


> Disadvantage:
> You need to parse and filter the big one session log to create
> user-specific log files which only conatin the sessions for that user.
> I have no experience with such tools and I think this filtering can
> not be 100% reliable

Exactly right; sftp-server doesn't include the username in most
messages so it's difficult to reliably associate messages with a user.


Kind regards

//Peter

[1] https://www.freedesktop.org/software/systemd/man/systemd.generator.html
[2] https://www.freedesktop.org/software/systemd/man/systemd.mount.html
[3] https://www.freedesktop.org/software/systemd/man/systemd.unit.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sftplog-generator.sh
Type: application/x-sh
Size: 633 bytes
Desc: not available
URL: <http://lists.mindrot.org/pipermail/openssh-unix-dev/attachments/20211001/ee327411/attachment-0001.sh>