Howto log multiple sftpd instances with their chroot shared via NFS

Wed Sep 29 01:00:12 AEST 2021

Hallo all, thank You all very much for your answers and suggestions.
Since the discussion has been shared between several mails now, I would like to try to summarize all up in one mail again here
and fill in the informations that where missing before.

We have 800 (eight hundred) sftp customers, each sftp customer has the same simple local Linux account on both sftp servers (simple entry in /etc/passwd and /etc/shadow etc.).
(Before we had only one sftp server, but for higher availability we want now to run two or more sftp servers parallel, accessed via TCP (sftp) load balancer)

Each customer account is in the group "sftp-cust", and only members of that group are allowed to login via sftp.
Each customer has it's chrooted home dir in /var/data/chroot/<username>/.

Here the (relevant part) of the sftpd config:

---------------------------------------------------------------
AllowGroups sftp-cust
Subsystem sftp internal-sftp -f LOCAL5 -l INFO
Match Group sftp-cust
    ChrootDirectory %h
    ForceCommand internal-sftp -u 0002 -f LOCAL5 -l INFO
    AllowTcpForwarding no
---------------------------------------------------------------

Each sftp customer's sftp activity needs to be available in a dedicated log file for each sftp customer, for our support to be able to look into it. Here is an example of this log file:

Sep 28 15:54:25 myhostname internal-sftp[1618]: session opened for local user <username> from [1.2.3.4]
Sep 28 15:55:52 myhostname internal-sftp[27918]: remove name "/in/file.dat"
Sep 28 15:55:52 myhostname internal-sftp[27918]: sent status No such file
Sep 28 15:55:52 myhostname internal-sftp[27918]: open "/in/file.dat" flags WRITE,CREATE,TRUNCATE mode 0666
Sep 28 15:55:52 myhostname internal-sftp[27918]: close "/in/file.dat" bytes read 0 written 2966
Sep 28 15:55:52 myhostname internal-sftp[27918]: set "/in/file.dat" modtime 20210928-15:55:46

If one does not use the /dev/log in the chroot environment (that is /var/data/chroot/<username>/dev/log absolute), you have a global sftpd log (I think in /var/log/messages on the server or something like that).
So a solution for the log problem would be, to not use the chroot logging, but to parse the global sftpd log (which is available on both sftpd servers locally for all local logins separated).
But I think this is not trivial to make this reliable and robust, since we need to parse the process id (sftpd session), check which username that session belongs to and write that log lines in the username specific log file.
Of course, process IDs are reused for different sessions, we have overlapping sessions etc. But if there is no other solution, than we need to try that. Maybe someone has already written such a session tracking parser.
Does somebody know of a log analyzer that can do this? A problem would be e.g. if the user never logs out, and the log file rotates daily, so after one day, the parser could not find any "session opened for local user xxx" log lines anymore.

If we want to try to keep using the available session chroot logging, man (8) sftp-server says:

"For logging to work, sftp-server must be able to access /dev/log.  Use of sftp-server in a chroot configuration therefore requires that
     syslogd(8) establish a logging socket inside the chroot directory."

As Peter has written here
https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-September/039669.html

as I understand it is hard coded in the system C library that the log devices name is /dev/log, so this cannot be changed (e.g. to log to chrooted /dev/log_hostname1 on hostname1 and chrooted /dev/log_hostname2 on hostname2).

With the syslog-ng we use, we only need to create the directory /var/data/chroot/<username>/dev for each user once we create a new account.

We have the following syslog-ng config snippet file for each sftp user:

source s_chroot_<username>    { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
destination d_sftp_<username> { file("/var/log/app/sftp/<username>.log"); };
log                           { source(s_chroot_<username>); destination(d_sftp_<username>); };

When starting syslog-ng the unix stream file /var/data/chroot/<username>/dev/log is created automatically by syslog-ng. If you delete it it gets recreated upon syslog-ng restart.

But the problem is that the last started syslog-ng aquires the lock for the NFS shared /var/data/chroot/<username>/dev/log so the other server cannot read it anymore and so there is no sftp log for the sessions on the other server.

I guess this is because syslog-ng creates /dev/log in the user's chroot directories as Unix stream socket (see https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.22/administration-guide/28#TOPIC-1209171).
 This seems also to be called IPC socket (inter-process communication socket)   or AF_UNIX socket.
"It is used in POSIX operating systems for inter-process communication. The correct standard POSIX term is POSIX Local IPC Sockets. Unix domain connections appear as byte streams, much like network connections, but all data _remains within the local computer_."

"It means that if you create a AF_UNIX socket on a NFS disk which is shared between two machines A and B, you cannot have a process on A writing data to the unix socket and a process on B reading data from that socket.
The communication happens at kernel level, and you can only transfer data among processes sitting in the same kernel."
(see source: https://stackoverflow.com/questions/783134/af-unix-domain-why-use-local-file-names-only )

Since we have 800 users, it would be impractical unrobust to use user-specifc e.g. bind) mounts (e.g. 800 bind-over-mounts). To keep it simple, clear and coherent, all user's homes must be on the same one singular NFS-Share.

We need to stick with Ubuntu Linux as we have established management process only for this operating systems.

I hope I did not forget some information that was missing before. Thanks  Hildegard

> Gesendet: Mittwoch, 22. September 2021 um 09:19 Uhr
> Von: "Damien Miller" <djm at mindrot.org>
> An: "Hildegard Meier" <daku8938 at gmx.de>
> Cc: openssh-unix-dev at mindrot.org
> Betreff: Re: Howto log multiple sftpd instances with their chroot shared via NFS
>
> On Tue, 21 Sep 2021, Hildegard Meier wrote:
>
> > OpenSSH 5.9p1 + 7.6p1
> >
> > syslog-ng 3.3.4 + 3.13.2
> >
> > Hello, having an Ubuntu server with sftpd running where /var/data/chroot/ is an NFS mount from a remote central NFS server,
> > and each sftpd user's chroot home is /var/data/chroot/<username>/
> > and every user has a log device /var/data/chroot/<username>/dev/log which I read in successfully with syslog-ng:
> >
> > source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
> > destination d_sftp_<username> { file("/var/log/sftp/<username>.log"); };
> > log { source(s_chroot_<username>); destination(d_sftp_<username>); };
> >
> > Now I have a second sftpd server in parallel, with the same user database and also mounts /var/data/chroot/ via NFS, and has the same syslog-ng config,
> > so every user can login on the one server or on the other. This is for high availability. This works so far.
> >
> > What is not working now is the sftpd logging: The sftp user's log is only available on one sftp server exclusively, and that is the one where syslog-ng was started least,
> > because as I understand it takes the exclusive unix socket file lock for each user's /dev/log.
> >
> > So, if a user logs in on the first server, where syslog-ng was started least, the user's sftp activity is logged on the first server.
> > But if the user logs in on the second server, it's sftp activity is not logged, neither on the second nor on the first server.
> >
> > If the syslog-ng is then restarted on the second server, the sftp user's activity is exclusively logged only on the second server and only for logins on the second server.
> >
> > How can I get the sftp user's activity be logged on each sftp server, when a user logs in to that server, while the user's home is shared on both servers via NFS?
>
> Right now there is no solution for this inside OpenSSH. There have been
> some proposals for post-auth logging to be proxied via the priviledged
> sshd monitor process but we haven't pursued them yet.
>
> Maybe someone with more Linux/NFS wit could suggest an OS-side solution
> for you?
>
> -d
>