Minimize sshd log clutter/spam from unauthenticated connections

[Carsten Andrich]

On 18.03.23 14:34, David Lang wrote:
> modern syslog daemons (including rsyslog, which is default on just 
> about every linux system) allow you to filter efficiently on the 
> message contents, not just the severity, so you can opt to throw out 
> the messages you don't want.
>
> I advocate for a slightly different way of dealing with it, filter 
> these messages from your main logstream, but put them into either a 
> script directly, or a separate file and have a script run against it. 
> Have the script report the number of these messgaes that you get in a 
> time period (minute, hour, whatever you want) and log that count back 
> into your log stream
>
> as Marcus Ranum said in his Artificial Ignorance writeup, the number 
> of times that an uninteresting thing happens can be interesting.
>
> If you see a big spike (or drop) is these attempts, it can indicate 
> cause for concern.

I run Debian with systemd-journald instead of rsyslog. AFAIK journald 
does not support filtering of its ingress log messages. Only the output 
can be filtered with journalctl, but by then it's already too late in 
terms of log spam on disk.

Regardless of which syslog solution is being used, I find it 
inconvenient to manually assemble and maintain a filter list. I believe 
an sshd configuration option, which of course defaults to false, would 
be the best solution. Currently, the sheer amount of messages from 
unauthenticated connections drowns out anything else. I took this 
opportunity to sift through sshd's messages of almost 2 years via:

journalctl -t sshd -o cat \
     | grep -v '^Accepted ' \
     | sed -E 's/[Uu]ser \S+/user .../' \
     | sed -E 's/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/0.0.0.0/' \
     | sed -E 's/port\s+[0-9]+/port 0/' \
     | sed -E 's/".*"/"..."/' \
     | sed -E 's/Change of username or service not allowed: .+/Change of username or service not allowed: .../' \
     | sed -E 's/Their offer: .+/Their offer: .../' \
     | sort -u

I found a select few attempts to mess with, identify, or exploit log 
parsing IDS/IPS software like fail2ban (and feel confirmed in my choice 
of an alternative solution with far less attack surface, see my other mail):

Invalid user $(ping -c 1 16e939dc.ad.xspzo.com) from ...
Invalid user ' $(ping -c 1 16e939dc.ad.xspzo.com) from ...
Invalid user ' or '1'='1' - from 176.100.42.41

Only two concerning messages came up:

error: beginning MaxStartups throttling
fatal: ssh_sandbox_violation: unexpected system call (arch:0xc000003e,syscall:20 @ 0x7f1d8469e1f5) [preauth]

I definitely wouldn't want to miss the above among an almost endless 
list of (invalid) usernames, (dis-)connecting IP addresses, invalid 
version strings, etc. that various script kiddies are playing with. 
While I do get your point, I doubt analyzing all of these individual 
messages has a reasonable cost-benefit ratio. I occasionally look at the 
amount of rejected connections (my nftables rules keep track of that) to 
see any unusual uptick of interest, upon which I investigate.

Best regards,
Carsten