[Twisted-Python] Nagle delays and twisted

[Brian Granger]

Hello all,

I have been implementing a custom protocol with twisted and have been  
having some odd performance problems.  My protocol consists of a TCP  
connection with multiple client/server commands.  For some types of  
commands, I found it took about 200 ms for the client and server to  
complete.  Because this was over loopback (which has a ping of 0.1  
ms), it seems very slow.  Because I am keeping the socket connected,  
I am not measuring the socket setup time.  Lucky for me, there was a  
discussion about this issue on comp.lang.python recently:

http://groups.google.com/group/comp.lang.python/browse_frm/thread/ 
507a711c61df7ad5/d6e871a24d8506a6#d6e871a24d8506a6

I guess the 200 ms delay is a well known effect of the Nagle  
Algorithm that TCP uses.  Supposedly, this delay can be eliminated by  
i) setting TCP_NODELAY on the socket or ii) avoiding sending multiple  
small messages without a response from the other end.

My understanding is that protocols that do the following will run  
into the problem:

C: short message 1
C: short message 2
S: reply for message 1 and 2

What is the preferred way of setting TCP_NODELAY on a socket  
controlled by twisted?  Are there any adverse side effects that this  
will have?

And second: does anyone have any tips for designing protocols that  
avoid this delay without setting TCP_NODELAY?  The obvious solution  
is to have a reply for each message:

C: short message 1
S: reply for 1
C: short message 2
S: reply for 2

But... this increases the latency of the protocol.  Any ideas or  
experiences with these issues would be greatly appreciated.

Thanks

Brian