[Twisted-Python] ENOBUF and Twisted

Jon Dyte jon at totient.demon.co.uk
Thu Aug 19 06:50:36 EDT 2004


This problem looks similar to this that some CORBA folk are seeing with 
Windows and large requests
http://www.omniorb-support.com/pipermail/omniorb-list/2004-August/025775.html

There are some links and suggested fixes in there, I'm not sure how relevant, 
but I thought it worth posting.


Jon


On Thursday 19 August 2004 07:48, screwtape at froup.com wrote:
> A colleague of mine was working with a server written in Twisted
> today, and he ran into a problem involving Twisted. He called me over
> (since I'm supposed to be the Python Guru here, which just means I'm
> slightly less ignorant than everybody else :) and we tracked the
> behaviour in question all the way down to twisted.internet.tcp.
>
> The situation is this: we have an upstream data provider who sends us
> data intermittently, all day. We have downstream clients on less
> reliable connections who want to recieve it. We have a sort of caching
> proxy in the middle that keeps track of all the data sent so far
> today, and every time a client connects we send them the total record
> of the day's events, and then send them new updates as they arrive.
>
> The problem occurred one day when we had a higher-than-usual amount of
> data incoming. The symptom was that any client attempting to connect
> would be immediately disconnected for No Obvious Reason. After some
> ferretting about through the codebase, we discovered that the
> 'writeSomeData' method of twisted.internet.tcp.Connection was getting
> an error it couldn't handle and dropping the connection.
>
> The swift insertion of a debugging printf showed that the error
> generated was ENOBUFS (Well, WSAENOBUFS, since it was running on
> Windows 2000), which made the Connection drop the, uh, connection
> immediately.
>
> Now, it occurs to me that ENOBUFS does not necessarily mean that the
> connection is irretrievably lost, it should be treated more like
> EWOULDBLOCK, except that buffers over a certain size will never, ever
> fit. I don't know whether special handling of ENOBUFS should happen in
> Twisted (to match its already-existing special handling of
> EWOULDBLOCK), or whether it should happen in the application (proper
> buffer-size autodetection might be a little intricate for a class like
> Connection that's supposed to be simple). But either way, just
> dropping the connection immediately is probably a Bad Thing.
>
> (for the record, the buffer we were trying to send was about 5MB, but
> I assume people trying to send, say CD or DVD disk images via
> twisted.web may well run into similar issues)
>
> Is there some clever way to get around this with the current version
> of Twisted? If I were to write a patch, how should I best attack the
> problem? Are there any other questions I should be asking? :)





More information about the Twisted-Python mailing list