Recent MAC improvements - stalls

[Chris Rapier]

I've been seeing intermittent stalls with the June 13th snapshot.
I've been able to recreate them pretty consistently and I've only been 
able to do it with the June 13th snapshot. I've included the level 1 
debug output. I hit enter around once a second to show the progress and 
where it stalls. This is happening when I am using the mac recycling 
code as a client side sink. It happens when I connect to a mac recycling 
server or a standard install. During the tests I ran over the last hour 
it would sometimes happen every other transfer and other times it would 
happen *maybe* one out of 15 times. This is happening with the default 
mac and the new umac-64

It *may* be due to the fact that we're running the default interface 
with a 9K MTU but its connecting to a 1.5K MTU source. I did try it over 
one of the 1.5K MTU NICs but saw the same problem. However, I'm not 
seeing these stalls with a std to std transfer.

I've no idea if this is a problem specific to my install on this machine 
or not. I'm thinking it is because I was unable to recreate the problem 
in the other direction. However, I thought you should be aware of it.


environment:

client - OpenSSH_4.6p1-snap20070613, OpenSSL 0.9.8b 04 May 2006

servers - OpenSSH_4.6p1-snap20070613, OpenSSL 0.9.7a Feb 19 2003
           OpenSSH_4.6p1, OpenSSL 0.9.7a Feb 19 2003

Host information - Linux brillo 2.6.20-web100 #6 SMP Wed Mar 21 14:35:36 
EDT 2007 i686 GNU/Linux - Xeon 2 CPU dual core at 2.8Ghz.

1Gb connection traversing 1 switch.
rtt min/avg/max/mdev = 0.054/0.108/0.171/0.035 ms
(syskonnect) skge eth2: Link is up at 1000 Mbps, full duplex,
                         flow control rx only
Machine is multihomed.

command:  ssh46-mac/scp -S ssh46-mac/ssh -oMacs=umac-64 at openssh.com 
-carcfour256 -v -P 42222 rapier at delta:~/2gb /dev/null

Executing: program ssh46-mac/ssh host delta, user rapier, command scp -v 
-f ~/2gb
OpenSSH_4.6p1-snap20070613, OpenSSL 0.9.8b 04 May 2006
debug1: Connecting to delta [128.182.66.67] port 42222.
debug1: Connection established.
debug1: identity file /home/pscnoc/.ssh/identity type -1
debug1: identity file /home/pscnoc/.ssh/id_rsa type -1
debug1: identity file /home/pscnoc/.ssh/id_dsa type 2
debug1: Remote protocol version 1.99, remote software version OpenSSH_4.6
debug1: match: OpenSSH_4.6 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.6
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client arcfour256 hmac-md5 none
debug1: kex: client->server arcfour256 hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<4096<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: checking without port identifier
debug1: Host 'delta' is known and matches the RSA host key.
debug1: Found key in /home/pscnoc/.ssh/known_hosts:5
debug1: found matching key w/out port
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: 
publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Trying private key: /home/pscnoc/.ssh/identity
debug1: Trying private key: /home/pscnoc/.ssh/id_rsa
debug1: Offering public key: /home/pscnoc/.ssh/id_dsa
debug1: Server accepts key: pkalg ssh-dss blen 433
debug1: read PEM private key done: type DSA
debug1: Authentication succeeded (publickey).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: Sending command: scp -v -f ~/2gb
Sending file modes: C0644 2147483648 2gb
Sink: C0644 2147483648 2gb
2gb 
                                                          2%   55MB 
54.9MB/s   00:36 ETA
2gb 
                                                          5%  110MB 
54.9MB/s   00:35 ETA
2gb 
                                                          8%  165MB 
55.0MB/s   00:34 ETA
2gb 
                                                         10%  220MB 
55.0MB/s   00:33 ETA
2gb 
                                                         13%  275MB 
55.0MB/s   00:32 ETA
2gb 
                                                         13%  276MB 
49.6MB/s   00:35 ETA
2gb 
                                                         13%  276MB 
44.6MB/s   00:39 ETA
2gb 
                                                         13%  276MB 
40.2MB/s   00:44 ETA
2gb 
                                                         13%  276MB 
32.5MB/s   00:54 ETA
2gb 
                                                         13%  276MB 
29.3MB/s   01:00 ETA
2gb 
                                                         13%  276MB 
26.4MB/s   01:07 ETA
2gb 
                                                         13%  276MB 
23.7MB/s   01:14 ETA
2gb 
                                                         13%  276MB 
21.4MB/s   01:22 ETA
2gb 
                                                         13%  276MB 
19.2MB/s   01:32 ETA
2gb 
                                                         13%  278MB 
17.5MB/s   01:41 ETA
2gb 
                                                         13%  278MB 
15.7MB/s   01:52 ETA