sftp performance problem, cured by TCP_NODELAY

[Rick Jones]

I ran a version (likely somewhat old) of what ships with HP-UX under a system 
call trace tool.  I'm not exactly sure what I was seeing, but I did see several 
instances of call flow like:

write four bytes to the socket
write lots of bytes to the socket  (<= 8KB IIRC)

which smells (an awful lot) like something writing a message length to a socket 
and then writing the rest of the message.

I also saw a pattern of

read four bytes from a socket (a different one)
read lots of bytes from a socket

Which seemed to support the header, data hypothesis

Now, the latter (read four read lots) is really just a question of CPU 
overheads, it does not affect what goes-out onto the network.

However, the former looks like a classic case of an "unfortuneate" (if being 
dimplomatic, "buggy" if not) coding practice that quite easily runs afoul of 
Nagle - writing "logically associated data" to the connection in separate calls.

Assuming my brief syscall trace analysis is correct and if that behaviour is the 
same in the contemporary versions of sftp, it might be better to fix that before 
asking to do TCP_NODELAY.  Writing the logically associated data to the 
transport will allow a smaller number of segments to be sent, saving CPU util, 
and still not run afoul of Nagle.  Setting TCP_NODELAY will only optimize (if we 
can even call it that) for time, and still send more segments than warranted.

rick jones