Subsystem sftp invoked even though forced command created

Wed Oct 25 08:52:57 AEDT 2023

Following up on this thread:

Splunk UF cannot be used to drive an external application as it uses a proprietary stream data format (over a TLS-protected channel), thus the only recipient of the data must be a Splunk server.

Three options have been presented to me:
* A standard HTTPS web server such as Apache, driven by curl on the client side
* Open Telemetry [https://opentelemetry.io/].
* Apache Nifi [https://nifi.apache.org/].

My suspicion is that there is a very deep, fundamental difference in one of the core libraries provided by Linux as compared to those provided by BSD or Sys V platforms (including Solaris, HP-UX, and AIX). None of the latter three exhibit the problem.

Mike McManus
Principal - Technology Security
GTO Security Governance Team - Unix
P: He/Him/His

AT&T Services, Inc.
20309 North Creek Pkwy, Bothell, WA 98011
michael.mcmanus at att.com  

From: openssh-unix-dev <openssh-unix-dev-bounces+mm1072=att.com at mindrot.org> On Behalf Of MCMANUS, MICHAEL P
Sent: Tuesday, September 19, 2023 7:59 AM
To: openssh-unix-dev at mindrot.org
Subject: RE: Subsystem sftp invoked even though forced command created

This is a new branch of an old thread, made necessary because the email system here purges sent messages after a period of time so I can't reply to the last message in the thread. The operative portion of that last message (retrieved from the archives and dated July 3, 2023) follows:

/*****/
So I set up a fresh key to use for this test, and gave it similar parameters.

I wasn't aware of the "restrict" argument but the original authorized_keys entries had a long list of restrictions:
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command="/.../receive.ksh" ssh-rsa ...

I did the same thing as I had done with the live key and got the same results (as expected).

It then occurred to me that the issue is probably on the server side, which did not change. So I set up the user and forced command on 3 lab servers:
OS		OpenSSH version
Ubuntu 18.04	OpenSSH_7.6p1 Ubuntu-4ubuntu0.7, OpenSSL 1.0.2n  7 Dec 2017
Solaris 11	OpenSSH_8.4p1, OpenSSL 1.0.2zf  21 Jun 2022
HP-UX 11.31	OpenSSH_8.1p1+sftpfilecontrol-v1.3-hpn14v20, OpenSSL 1.1.1d  10 Sep 2019

I got the following results:
OS		ssh					sftp					SSH_ORIGINAL_COMMAND
Ubuntu 18.04	Hung waiting for input, pressed ^C	Obtained sftp prompt. 			Not logged
Solaris 11	Hung waiting for input, pressed ^C	Hung waiting for input, pressed ^C	Command: internal-sftp
HP-UX 11.31	Hung waiting for input, pressed ^C	Hung waiting for input, pressed ^C	Command: /opt/ssh/libexec/sftp-server

So the issue looks like it happens across Linux distributions, but not on non-Linux. Curiouser and curiouser.

I then changed the forced command to /usr/bin/logger and got similar results. I think the default priority for logs on our servers goes nowhere; next round I'll have to specify a priority that is actually logged.
/*****/

The conclusion I came to was that the Linux distributors are working with a modified version of the Berkeley sources (they have to, in order to back-port security fixes like they do). Therefore I downloaded OpenSSH 9.4p1 sources from the Berkeley depot and compiled them on a Rocky Linux 8 server. 

Sadly, the problem remains. I noticed in the logs there were references to Red Hat specific configuration, but that was on the client side and not the server side. I'm beginning to think that there is something in the configuration of OpenSSH on Linux servers which prioritizes the sftp subsystem over the forced command, whether the binaries are downloaded from the OS distributor or built from sources. 

Unless I can get some expert advice, I believe this kills off the use of SSH as a quick and secure transport for unattended file transfer and ingestion by applications, at least on Linux. It will have to be replaced by TLS or the Splunk Universal Agent. Pity, that.

Mike McManus
Principal - Technology Security
GTO Security Governance Team - Unix
P: He/Him/His

AT&T Services, Inc.
20205 North Creek Pkwy, Bothell, WA 98011
mailto:michael.mcmanus at att.com  

-----Original Message-----
From: openssh-unix-dev <mailto:openssh-unix-dev-bounces+mm1072=att.com at mindrot.org> On Behalf Of MCMANUS, MICHAEL P
Sent: Thursday, June 29, 2023 3:06 PM
To: mailto:openssh-unix-dev at mindrot.org
Subject: Subsystem sftp invoked even though forced command created

*** Security Advisory: This Message Originated Outside of AT&T ***.
Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.

Folks,

I'm curious if the documented behavior of portable OpenSSH (specifically Linux) may be at odds with the actual behavior I have seen in my experiments. Here is the background:

I manage an application which collects data from a client script (Korn shell) which runs on Unix and Linux servers across the entire enterprise. The client communicates with a Linux server (currently running RHEL 7.9, reporting OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017) using a technique we call SSH Pass-Through and described in an article in ;login: (Secure Automated File Transfer, vol. 30, no. 4, August 2005, by Mark McCullough who was also the original author of the client). The technique is predicated on using a forced command, which we specify per key with the "command=" directive in the authorized_keys file. (The user ID we use performs several functions, so we cannot use ForceCommand in configuration.) 

In theory, any SSH connection made to the account with the corresponding private key should be forced to run the command provided, which accepts a fully-qualified domain name, environment, and stream of XML data on standard input and stores the data in a file with a name reflecting the FQDN and environment along with a date stamp. In all documentation I have to hand, the forced command is applied to all sessions including interactive, non-interactive, and *subsystem calls*.

An authorized penetration tester brought to my attention that the private key embedded in the application can be extracted and used to launch a WinSCP session against the user ID which the client uses to send the data to the server. This session has the same access to all files on the server which the original user ID does, including the ability to traverse directories. This should not happen if the documentation is correct.

To test the theory, I've slightly altered the forced command to output log data to indicate whether the forced command is even executed, and if so, what command line it was passed from the client (or WinSCP). The added code is as follows:

LOGFILE=/tmp/name-of-file.log
if [[ $SSH_ORIGINAL_COMMAND ]]; then
    print "Command: $SSH_ORIGINAL_COMMAND" >> $LOGFILE
else
    print "No SSH_ORIGINAL COMMAND set" >> $LOGFILE
fi

I ran the client as is and received the following entry in the log:
Command: 2>/dev/null

I then set a flag in the client which removes the redirection to /dev/null and received:
No SSH_ORIGINAL COMMAND set

Finally, I connected via WinSCP as instructed by the pen tester and received no further data in the log. This indicates to me that the sftp subsystem does not even execute the forced command.

The subsystem configuration in sshd_config is 
# override default of no subsystems
Subsystem sftp  /usr/libexec/openssh/sftp-server

I would almost expect this sort of behavior with the internal-sftp subsystem, but not with an external executable. Can anyone shed some light on this situation?

Mike McManus
Principal - Technology Security
GTO Security Governance Team - Unix
P: He/Him/His

AT&T Services, Inc.
20205 North Creek Pkwy, Bothell, WA 98011
mailto:michael.mcmanus at att.com  

_______________________________________________
openssh-unix-dev mailing list
mailto:openssh-unix-dev at mindrot.org
https://urldefense.com/v3/__https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev__;!!BhdT!gAfqyQFKV_nTGhaz-PwBqHZat18A67O51MZnz1_jFaordOKkpGy9dSyhdHYW0dqNiT5SUd2Mh6M$
_______________________________________________
openssh-unix-dev mailing list
mailto:openssh-unix-dev at mindrot.org
https://urldefense.com/v3/__https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev__;!!BhdT!jB623mgetR7HfBahE8oix3S2IvTlS8gi8b9cEjavaF6wUCmbu2aiTNkro69I10HFjUhp8XhV14A$