[Twisted-Python] ENOBUF and Twisted

Jon Dyte jon at totient.demon.co.uk
Thu Aug 19 04:50:36 MDT 2004

This problem looks similar to this that some CORBA folk are seeing with 
Windows and large requests

There are some links and suggested fixes in there, I'm not sure how relevant, 
but I thought it worth posting.


On Thursday 19 August 2004 07:48, screwtape at froup.com wrote:
> A colleague of mine was working with a server written in Twisted
> today, and he ran into a problem involving Twisted. He called me over
> (since I'm supposed to be the Python Guru here, which just means I'm
> slightly less ignorant than everybody else :) and we tracked the
> behaviour in question all the way down to twisted.internet.tcp.
> The situation is this: we have an upstream data provider who sends us
> data intermittently, all day. We have downstream clients on less
> reliable connections who want to recieve it. We have a sort of caching
> proxy in the middle that keeps track of all the data sent so far
> today, and every time a client connects we send them the total record
> of the day's events, and then send them new updates as they arrive.
> The problem occurred one day when we had a higher-than-usual amount of
> data incoming. The symptom was that any client attempting to connect
> would be immediately disconnected for No Obvious Reason. After some
> ferretting about through the codebase, we discovered that the
> 'writeSomeData' method of twisted.internet.tcp.Connection was getting
> an error it couldn't handle and dropping the connection.
> The swift insertion of a debugging printf showed that the error
> generated was ENOBUFS (Well, WSAENOBUFS, since it was running on
> Windows 2000), which made the Connection drop the, uh, connection
> immediately.
> Now, it occurs to me that ENOBUFS does not necessarily mean that the
> connection is irretrievably lost, it should be treated more like
> EWOULDBLOCK, except that buffers over a certain size will never, ever
> fit. I don't know whether special handling of ENOBUFS should happen in
> Twisted (to match its already-existing special handling of
> EWOULDBLOCK), or whether it should happen in the application (proper
> buffer-size autodetection might be a little intricate for a class like
> Connection that's supposed to be simple). But either way, just
> dropping the connection immediately is probably a Bad Thing.
> (for the record, the buffer we were trying to send was about 5MB, but
> I assume people trying to send, say CD or DVD disk images via
> twisted.web may well run into similar issues)
> Is there some clever way to get around this with the current version
> of Twisted? If I were to write a patch, how should I best attack the
> problem? Are there any other questions I should be asking? :)

More information about the Twisted-Python mailing list