Howto log multiple sftpd instances with their chroot shared via NFS

[Peter Stuge]

Hallo Hildegard,

thanks for the summary.

Hildegard Meier wrote:
> ---------------------------------------------------------------
> AllowGroups sftp-cust
> Subsystem sftp internal-sftp -f LOCAL5 -l INFO
> Match Group sftp-cust
>     ChrootDirectory %h
>     ForceCommand internal-sftp -u 0002 -f LOCAL5 -l INFO
>     AllowTcpForwarding no
> ---------------------------------------------------------------

This is very sensible configuration but it unfortunately makes the
LD_PRELOAD approach a bit less desirable. It will still work though.


> So a solution for the log problem would be, to not use the chroot
> logging, but to parse the global sftpd log
..
> But I think this is not trivial to make this reliable and robust,

Agreed. If going the log analysis path I would add -e to the
internal-sftp command line and run sshd with stderr connected to that
parser, to be able to keep track of events in a sensible way.
Log rotation would happen only after (or in) the parser.


> If we want to try to keep using the available session chroot
> logging, man (8) sftp-server says:
> 
> "For logging to work, sftp-server must be able to access /dev/log.  Use of sftp-server in a chroot configuration therefore requires that
>      syslogd(8) establish a logging socket inside the chroot directory."
> 
> As Peter has written here
> https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-September/039669.html
> 
> as I understand it is hard coded in the system C library that the log
> devices name is /dev/log, so this cannot be changed (e.g. to log to
> chrooted /dev/log_hostname1 on hostname1 and chrooted /dev/log_hostname2
> on hostname2).

Using LD_PRELOAD e.g. with the code I attached to that email allows
replacing the C library syslog functionality. My attachment does
indeed log to /dev/log_hostname1 on hostname1 instead of /dev/log.

However because of internal-sftp (which I still prefer over the alternative;
maintaining an sftp-server binary and possibly libraries in every homedir)
the entire sshd would have to be run with LD_PRELOAD.

Note that LD_PRELOAD is only neccessary on n-1 servers; the first
server can still use /dev/log and the full C library syslog() code.

If the second server where LD_PRELOAD is used does not need to
support anything besides SFTP, or if you can mandate that, then
it could be okay to run the entire sshd with LD_PRELOAD. Your IT
security colleagues may well object and they're not wrong; it's not
much code but still quite a mighty hammer.


> > Maybe someone with more Linux/NFS wit could suggest an OS-side solution
> > for you?

Before going down the LD_PRELOAD path I considered an overlayfs solution.

It would be nice, but the writable (upper) directory can't be NFS
with overlayfs so it can not work on the SFTP servers.

But using overlayfs can work on the NFS server. There would then be
one export for each SFTP server, all of them reaching the same
homedir contents except for different (empty) <homedir>/dev/ subdirs.

This requires maintaining a second homedir structure on the NFS server,
but that structure only needs to contain a homedir with one empty dev
subdir under and nothing else, so it may be doable. Homedirs need
correct permissions and only one of the two directory inodes for the
homedir itself will be changed by the respective SFTP server, but on
the plus side it requires only a single overlay mount, nothing per-user
and no LD_PRELOAD stuff on the SFTP servers.


On the NFS server, assuming that /home is where real homedirs exist
(this gets mounted directly by one SFTP server), that /var/home2 is
the second homedir structure with only homedirs and an empty dev subdir
in each, that /var/home2_work is an empty directory on the same filesystem
as /var/home2 and that /nfs/home2 is the second export mounted by the
second SFTP server, then set it up like so:

mount -t overlay overlay -olowerdir=/home,upperdir=/var/home2,workdir=/var/home2_work /nfs/home2


Then export /home and /nfs/home2 via NFS.

The SFTP server mouting /home will access /home/<user>/dev/log
(both syslog-ng and internal-sftp) and the SFTP server mouting
/nfs/home2 will instead access /var/home2/<user>/dev/log but
everything else within the homedir is to and from /home/<user>.

If this change can be done on the NFS server I think it's what I'd prefer.


Kind regards

//Peter