[Bug 3905] New: Deadlock when channel window adjust is stuck behind sftp read requests

[bugzilla-daemon at mindrot.org]

https://bugzilla.mindrot.org/show_bug.cgi?id=3905

            Bug ID: 3905
           Summary: Deadlock when channel window adjust is stuck behind
                    sftp read requests
           Product: Portable OpenSSH
           Version: 10.2p1
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: P5
         Component: sftp
          Assignee: unassigned-bugs at mindrot.org
          Reporter: matt at ucc.asn.au

Overview:
OpenSSH sftp client sends up to 64 pipelined read requests (each 32kB).
If they are all enqueued to a sftp server before enough read response
is sent, the total amount of outstanding response data will exceed the
channel send window. The client will send window adjust messages as it
receives responses, but those will be enqueued after the read requests,
so be handled by the server before it runs out of window. The result is
a deadlock.

I'd be interested in ideas on how to both avoid it from my
implementation (to deal with existing OpenSSH sftp clients) and how
OpenSSH could be modified to avoid it without losing the benefit of
pipelining the requests. 

Steps to reproduce:
(sorry this isn't straightforward, I can try reproduce something with a
modified OpenSSH if that's easier, let me know if you want that).
I've been testing using an experimental ssh/sftp server (sunset). It
has only a 512 byte send buffer (intended for microcontrollers).

I think it could probably also happen with more normal implementations
if there is a very slow network connection (or perhaps asymmetrically
slow in the server->client direction).

- Testing on Linux, requires a rust toolchain.
- Fetch tag https://github.com/mkj/sunset/tree/matt/sftp-debug-20251214
- Add a usable key to localhost authorized_keys for testing
- cd demo/sftp/std/testing; ./log_get_single_long.sh
It will run for a while, eventually it will get stuck.

Expected results:
Completes the transfer

Actual results:
Once the difference between outstanding requests and received responses
reaches around 2048kB, it will stop. After adding additional logging to
the ssh server, the channel window is seen to get to 0.

I've confirmed that the client and server both are subtracting the same
channel data amounts for the channel window.

The end of the sftp client log where it gets stuck:
[2025-12-14T14:29:50.389239378Z] debug3: Received reply T:103 I:148
R:64
[2025-12-14T14:29:50.392080974Z] debug3: Received data 4685824 ->
4718591
[2025-12-14T14:29:50.394617295Z] debug3: Request range 6782976 ->
6815743 (63/64)
[2025-12-14T14:29:50.511720459Z] debug2: channel 0: window 1998526 sent
adjust 98626
[2025-12-14T14:29:50.514963795Z] debug3: Received reply T:103 I:149
R:64
[2025-12-14T14:29:50.517640340Z] debug3: Received data 4718592 ->
4751359
[2025-12-14T14:29:50.521306634Z] debug3: Request range 6815744 ->
6848511 (63/64)

Some discussion is in
https://github.com/mkj/sunset/pull/29#issuecomment-3650761405 (just for
reference, probably not much to read that isn't described here).

-- 
You are receiving this mail because:
You are watching the assignee of the bug.