[Twisted-Python] ENOBUF and Twisted

Andrew Bennetts andrew-twisted at puzzling.org
Thu Aug 19 09:02:02 EDT 2004


On Thu, Aug 19, 2004 at 10:01:59PM +1000, Tim Allen wrote:
> On 19 Aug 2004, at 19:10, Andrew Bennetts wrote:
> >>screwtape at froup.com wrote:
> >>>A colleague of mine was working with a server written in Twisted
> >>>today, and he ran into a problem involving Twisted. He called me over
> >>>(since I'm supposed to be the Python Guru here, which just means I'm
> >>>slightly less ignorant than everybody else :) and we tracked the
> >>>behaviour in question all the way down to twisted.internet.tcp.
> >I wonder if the win32reactor or iocpreactor get more useful 
> >information than
> >select and so can handle this better?
> 
> It's not so much that the default select reactor makes information 
> unavailable.. as far as I can tell, the Python socket object is 
> correctly raising ENOBUFS, and t.i.tcp.Connection is very deliberately 
> throwing that information away rather than handling properly.

Well, the problem is that currently the select reactor assumes that if
select says a socket is writeable, then it is writeable, i.e it is safe to
attempt to write any amount of data to it (with the understanding that being
non-blocking it might only be able to accept some of that data immediately).
Apparently this assumption is not quite accurate.

So far in this thread I've seen two theories about what the exact problem
is, and I think the correct fix depends on which is actually the cause.

Theory 1: Writing large (i.e. 5MB) chunks to sockets in a single send call
will always fail, and should.  The fix for this would be to make sure that
no more than 64kB (or whatever) is passed to send at a time, regardless of
what is passed to transport.write.

Theory 2: Writing lots and lots of data very fast can cause the network
hardware's/device driver's internal buffers to fill up, thus making it
impossible to write even though select claims it is.  If this is the case,
the socket needs to be ignored for a moment to give those buffers a chance
to flush.

A test case proving which is actually the case would be most welcome :)

-Andrew.





More information about the Twisted-Python mailing list