Howto log multiple sftpd instances with their chroot shared via NFS

[Hildegard Meier]

Hello all, first let me thank you all for your input and suggestions.

I think I have found now a workaround for the problem.

> Since we have 800 users, it would be impractical unrobust to use user-specifc e.g. bind) mounts (e.g. 800 bind-over-mounts). To keep it simple, clear and coherent, all user's homes must be on the same one singular NFS-Share.

It seems, that I cannot uphold this requirement of not having user-specific bind-over-mounts. (while the requirement that there is only exactly one nfs mount is still true).

My workaround is the following:

sudo mkdir /var/data/dev    # Create a directroy under which user subdirectories are created

For very username <username> and the user's primary group <groupname> do the following:

sudo mkdir             /var/data/dev/<username>
sudo chmod 550         /var/data/dev/<username>       # This restrictive permission is a requirement I think
sudo chgrp <groupname> /var/data/dev/<username>       # so the user can read the directory

So the new directory is exactly the same as the existing /var/data/chroot/<username>/dev directory (which is on the nfs mount /var/data/chroot/).

Then do
mount --bind /var/data/dev/<username>     /var/data/chroot/<username>/dev

so /var/data/chroot/<username>/dev is now effectively local on the sftp server, not anymore on nfs mount.

Then change the syslog-ng config

from
source s_chroot_<username>     { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };

to
source s_chroot_<username>     { unix-stream("/var/data/dev/<username>/log" optional(yes) ); };

(this is not strictly needed, but I think it's nice having syslog-ng definitely now only reading from local file, guaranteed not from nfs mount anymore)

I have tested this successfully with one user so far - whether the user logs in on the one sftp server or the other, syslog-ng can now log the sftp session on the affected sftp server.

While it is still a mess to have 800 (or in the future maybe 2000 or 3000) bind mounts which I very much would like to avoid, I think this workaround is acceptable because the impact is limited to the logging functionality and not the sftp service itself. Speaking, if there are problems with the bind mounts, the impact is only that there is no sftp session logging.
Also, this is very simple, uniform and clear.
And one has 100% reliable sftp session logging.
No need to change anything else of the well established production system.

I think I will create systemd unit files for every bind mount (I experienced systemd dependency cycles with bind mounts in /etc/fstab on Ubuntu18 boot before anyway, so systemd units are needed anyway).
Maybe one can define a systemd dependency so that when you unmount the nfs mount /var/data/chroot/, all the bind mounts under it are automatically unmounted before (sftpd needs to be stopped first of course, otherwise mount would be busy), and also the bind-mounts automatically be mounted after /var/dat/chroot is mounted again.

So there should not be much hassle with all those simple uniform bind mounts.

I think I will implement a nagios monitoring check that checks if for every username /var/data/chroot/<username>/dev is a mountpoint, so logging is assured.

According to https://serverfault.com/questions/102588/maximum-numbers-for-file-system-mounts-in-linux/927860#927860 the Linux Kernel mount maximum is 100 000 so hopefully 2000 or 3000 will not be a problem.

--
Possible alternative
I guess (not tested) an alternative could be to hardlink (so needs to be on the same (nfs) file system) every /var/data/chroot/<username>/dev/log to e.g. /var/data/chroot/general_dev_log  (which would need access permission for all users of course)
and do
mount --bind /var/data/general_dev_log /var/data/chroot/general_dev_log

so every sftp session would log to the one and only log device (unix socket), again using the local bind mount.
So you could write all the sftp session logs of all users into one log file with syslog-ng.
Advantage:
- only one bind mount needed
- only one syslog-ng unix-stream configuration needed

Disadvantage:
You need to parse and filter the big one session log to create user-specific log files which only conatin the sessions for that user.
I have no experience with such tools and I think this filtering can not be 100% reliable, there will always be some cases where the filter will fail and log entries will be missing.

Best regards