Minimize sshd log clutter/spam from unauthenticated connections
carsten.andrich at tu-ilmenau.de
Sun Mar 19 04:16:44 AEDT 2023
On 18.03.23 14:34, David Lang wrote:
> modern syslog daemons (including rsyslog, which is default on just
> about every linux system) allow you to filter efficiently on the
> message contents, not just the severity, so you can opt to throw out
> the messages you don't want.
> I advocate for a slightly different way of dealing with it, filter
> these messages from your main logstream, but put them into either a
> script directly, or a separate file and have a script run against it.
> Have the script report the number of these messgaes that you get in a
> time period (minute, hour, whatever you want) and log that count back
> into your log stream
> as Marcus Ranum said in his Artificial Ignorance writeup, the number
> of times that an uninteresting thing happens can be interesting.
> If you see a big spike (or drop) is these attempts, it can indicate
> cause for concern.
I run Debian with systemd-journald instead of rsyslog. AFAIK journald
does not support filtering of its ingress log messages. Only the output
can be filtered with journalctl, but by then it's already too late in
terms of log spam on disk.
Regardless of which syslog solution is being used, I find it
inconvenient to manually assemble and maintain a filter list. I believe
an sshd configuration option, which of course defaults to false, would
be the best solution. Currently, the sheer amount of messages from
unauthenticated connections drowns out anything else. I took this
opportunity to sift through sshd's messages of almost 2 years via:
journalctl -t sshd -o cat \
| grep -v '^Accepted ' \
| sed -E 's/[Uu]ser \S+/user .../' \
| sed -E 's/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/0.0.0.0/' \
| sed -E 's/port\s+[0-9]+/port 0/' \
| sed -E 's/".*"/"..."/' \
| sed -E 's/Change of username or service not allowed: .+/Change of username or service not allowed: .../' \
| sed -E 's/Their offer: .+/Their offer: .../' \
| sort -u
I found a select few attempts to mess with, identify, or exploit log
parsing IDS/IPS software like fail2ban (and feel confirmed in my choice
of an alternative solution with far less attack surface, see my other mail):
Invalid user $(ping -c 1 16e939dc.ad.xspzo.com) from ...
Invalid user ' $(ping -c 1 16e939dc.ad.xspzo.com) from ...
Invalid user ' or '1'='1' - from 22.214.171.124
Only two concerning messages came up:
error: beginning MaxStartups throttling
fatal: ssh_sandbox_violation: unexpected system call (arch:0xc000003e,syscall:20 @ 0x7f1d8469e1f5) [preauth]
I definitely wouldn't want to miss the above among an almost endless
list of (invalid) usernames, (dis-)connecting IP addresses, invalid
version strings, etc. that various script kiddies are playing with.
While I do get your point, I doubt analyzing all of these individual
messages has a reasonable cost-benefit ratio. I occasionally look at the
amount of rejected connections (my nftables rules keep track of that) to
see any unusual uptick of interest, upon which I investigate.
More information about the openssh-unix-dev