Memory allocation gone awry with OpenSSH 3.(3,4)p1

[Mihnea-Costin Grigore]

     Hello,

     I have recently upgraded from an older version of OpenSSH to the
newer 3.3p1 (first) and now to 3.4p1 - because of the discovered
vulnerabilities. The default setting for these versions is to use
privilege separation, and this seems to trigger some weird bug with my
systems, causing sshd to consume memory until it barfs.
     Details:

1. OpenSSH 3.3p1, privsep on, compression on: after authenticating
someone, consumes all the available memory, eventually is killed by the
kernel.
   If privsep or compression are off, everything is working fine; also,
when logging in with another client (putty instead of ssh from the same
release) everything seems to be working fine (could not get it to crash
with putty).

2. OpenSSH 3.4p1 - same behaviour, except now there is a limit to the
memory sshd tries to alloc, and it just dies with
"fatal: buffer_append_space: alloc 10506240 not supported"
anytime privsep is on and compression is on.

     After some digging, we were able to trace the problem to
"buffer_uncompress" in compress.c - it seems that in some circumstances
the inflate call returns Z_OK and incoming_stream.avail_out == 0, thus
the program alloc's another sizeof(buf)-0 = 4096 bytes for the
output_buffer, then repeats the call to inflate, which again returns
status==0 and avail_out==0, thus entering an infinite loop.
     This is what happens under strace:

=====[cut here]=====
[pid  8379] old_mmap(NULL, 151552, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x404cc000
[pid  8379] mremap(0x404cc000, 151552, 188416, MREMAP_MAYMOVE) =
0x404cc000
[pid  8379] mremap(0x404cc000, 188416, 225280, MREMAP_MAYMOVE) =
0x404cc000
[pid  8379] mremap(0x404cc000, 225280, 262144, MREMAP_MAYMOVE) =
0x404cc000
[pid  8379] mremap(0x404cc000, 262144, 299008, MREMAP_MAYMOVE) =
0x404cc000
[pid  8379] mremap(0x404cc000, 299008, 335872, MREMAP_MAYMOVE) =
0x404cc000
[... and so on, until the limit is reached or memory is full ...]
=====[and here]=====

     Note the 151552 value, which is always the same when the program
crashes and never appears when it doesn't...

     All of this leds me to believe that there is a bug in the "custom"
memory allocating functions passed to zlib's inflate, and it is only
triggered in some (peculiar) circumtances.

     The bug was reproduced on the following systems:

1. Linux 2.4.19-pre8 kernel, glibc 2.2.0, gcc 2.95.3, zlib 1.1.3
2. Linux 2.4.18 kernel, glibc 2.2.0, gcc 2.95.3, zlib 1.1.4

     It was not possible to reproduce the bug on these systems:

1. Linux 2.4.19-pre9-ac2 kernel, glibc 2.2.5, gcc 3.0.4, zlib 1.1.4
2. Linux 2.4.17-ac3 kernel, glibc 2.2.0, gcc 2.95.2/egcs 1.1.2, zlib 1.1.4

     In all the configurations the same compile-options where used, with
and without optimizations, without any effect on the results.

     Hope we can solve this somehow... I can provide additional data/tests
if required.

Regards,

--
Mihnea-Costin Grigore          [ "Tenebus Ipsilo Ibinem Catehens" ]
E-mail: mgc8 at totalnet.ro       Home Page: http://mgc8.virtualave.net