Followup on Inquiry about regreSSHion postmortem

[Rene Malmgren]

Before I say anything, about the matter at hand I would like to comment on a few general things. I have been using ssh for about 30 years. OpenSSH has been the implementation that I have used by far the most, so its and understatement that I take not pleasure in the statement below. But on the other hand, cybersecurity is important to me, and it is totally unacceptable to be in a situation where backdoors a purposely introduced into the most critical software our society depends on by its lead developer, and nothing happens.

First of all, it is incorrect to say (as Demian does below) that I am accusing anybody of anything, I am not a prosecutor so it's not my role to accuse people of things. What I do is that I go over code and make assessments of what has happened and from that make recommendation about actions our customers should take to avoid being targeted by attackers in the future, I work in the crypto industry so for us security matters, so this should be no small matter.

I made a post mortem on CVE-2024-6387 when it was released and my clear recommendations based on the evidence found in the gitlogs was (and still is): decommission and replace.  Actually, looking at how the open-ssh community has handled what has happened the recommendation is if anything on more firm ground, more on that below.

I have made a rewrite of the original recommendation that was in a PDF on my (private) blog, if anybody is interested.

https://againstallflags.wordpress.com/2025/08/05/regrettable-regresshion/

With this out of the way let me clarify a few things about the assesment.

It's incorrect to say (as Demian Miller does below) that the assessment states that it's my conclusion that two lines (the two critical lines) where accidentally dropped during a merge indicate ill intent. The situation is considerably more serious, and I understand why Damien avoids to include the link to the evidence in his "reply".

Actually, there is no evidence in the available data that such a merge even has happened, instead the most probable scenario is that this is (yet another) incorrect statement from Damien Miller to cover up what actually happened, in futile hope that this is going to go away...

Instead, the assessment is based on the following established facts, see blogpost above for the details,

1. The feature (line numbers in log messages) introduced is of questionable value to the end-user, less than 0.1% of the users will use it.
2. The way it is implemented is very very complicated, from an architectural point of view it's really really ugly.
3. There is no obvious reason to choose this implementation if the goal is just to implement the feature.
4. It can't be explained away with some intent to redesign the logging framework, and of follow-up work has been seen.
5. The two lines in question can only have been deleted on purpose by the commiter, not as a result of a reasonable merge conflict, or any other "honest" mistake.
6. A notable side effect (post mortem) of the very very complected rewrite is that it hides the deletion of the two lines in a 500+ lines diff, while providing plausible deniability to the committer. "Anybody can make a mistake".
7. It was done by the lead developer on OpenSSH, who has full knowledge of CVE-2006-5051 and ssh:s internals.

Therefor the conclusion is that the following scenario is the most probable (above 90%):

1. The commit was designed on purpose in such a way as to hide the intentional reintroduction of CVE-2006-5051.
2. This "feature" is part of the smokescreen.
3. The overly complicated design is not a bug; it's a feature to hide a reintroduction of a bug.

The following facts makes it even sadder:

1. More than one year has passed.
2. Nobody from the OpenSSH organization has managed to make an independent investigation into CVE-2024-6387.
3. Even when the evidence has been pointed out to them, the action is not to look at the evidence and take action, instead we get more smokescreens, more head in sand...

I will leave out comment about how dangerous this vulnerability is, since this part of the discussion is super Flavious, if the lead developer decides to go to this length to introduce a CVSS Score 8+ backdoor into the code on purpose, then there are other and more valuable discussions to be had.

with full respect
--Rene

Below is a copy of the email that appeared on  openssh-unix-dev

----------------------------------------------------------------------------------------------

If I'm to be accused of deliberately or incompetently inserting bugs
into OpenSSH, I'd prefer the conversation happen in public.

On Thu, 7 Aug 2025, Aaron Rainbolt wrote:

> Hi Damien,
>
> You probably remember me from the thread about post-quantum-secure
> signature algorithms for openSSH. I'm a contributor to the Kicksecure
> and Whonix projects, two Debian derivatives that attempt to provide
> above-average security by hardening configuration, improving isolation
> between privileged and unprivileged operations, and educating users on
> how to avoid making common mistakes that can lead to compromises or
> increase attack surface. Since we're firmly in the "paranoid security"
> community, we occasionally get feedback from users that may seem a bit
> odd at first glance, and some of said odd feedback is what led to this
> email.
>
> A user who wishes to remain anonymous sent us a document arguing that
> the way in which regreSSHion (CVE-2024-6387) was introduced was, in the
> user's words, "unreasonably careless". They appear to have looked
> through the OpenSSH Git commit history to analyze the history of
> CVE-2006-5051 and the surrounding code, and how it relates to
> CVE-2024-6387, then made their conclusion based on how the code changed
> over the years. The document references a blog post written by René
> Malmgren (who is CC'd on this email), who argues that the vulnerability
> was likely "reintroduced intentionally", using a much shorter analysis
> of some of the same Git commits. The user did not entirely agree with
> René's assessment, but still found some of the circumstances
> surrounding regreSSHion to be worrying enough that they contacted us
> about it.
>
> Given our focus on security, we obviously want to act on such feedback
> if (and only if) it is reasonable. But given our not-very-strong
> understanding of OpenSSH's internals, lack of familiarity with the
> project's development procedures, and positive statements from others
> about OpenSSH's internal design and security, we aren't really sure
> what to make of this. Both of the writeups are likely missing context
> since they're based mostly on Git commits and commit messages, and
> neither individual appears to have any familiarity with OpenBSD
> development practices. I talked about this with the lead Kicksecure
> developer ("adrelanos", who is also CC'd on this email), and we decided
> it would probably be a good idea to forward the documents to you in
> case you wanted to comment on them. I've attached the document to this
> email.

The regreSSHion bug was my mistake.

How was the bug introduced? In a commit (752250caa) that improved
OpenSSH's logging framework, a portable-OpenSSH specific #ifdef was
incorrectly dropped.

How did this happen? It's the reason that Occam's Razor would
suggest - two portable-specific lines got lost in a ~500 line diff
due to a human error (mine) while merging.

How did the human error go unnoticed? Essentially all non-trivial
changes to OpenSSH are reviewed before being committed, but the merge
process is not reviewed again. Mistakes made in manually merging
changes with conflicts can go unnoticed.

Could we do better here? With more resources, sure. The critics you
cite can help here by looking themselves. The commits are all public,
and attributed and we provide a 1:1 mapping to originating OpenBSD
commits via the OpenBSD-Commit-ID marker in the commit message.
Finding other mistakes of this nature should be simple for them,
should they exist. I invite them to look.

How did we respond to this problem? By improving the architecture of
sshd and introducing controls that make exploitation of any non-
deterministic bug significantly harder to exploit. It's IMO weird
that your anonymous accuser treats this work as somehow evidence
of further incompetence.

Was this a intentional backdoor? Obviously a denial would be
redundant here, but if it was a backdoor then it would be a pretty
ineffective one. The attack has a low probability of success even
under ideal circumstances, taking many hours and being extremely
noisy in the process. A year after discovery, I'm still not aware
that anybody has make a successful exploit for 64-bit systems.

Compare this to actual backdoors, such as Dual_EC_DRBG or the xz
incident and you can see that it's missing two critical features:
1) determinism (as mentioned above) and 2) NOBUS - "no-one but us",
the property that the backdoor is only usable by its originators.

Draw your own conclusions from this.

-d