[Twisted-Python] ENOBUF and Twisted
exarkun at divmod.com
Sat Aug 21 01:50:39 MDT 2004
screwtape at froup.com wrote:
> A colleague of mine was working with a server written in Twisted
> today, and he ran into a problem involving Twisted. He called me over
> (since I'm supposed to be the Python Guru here, which just means I'm
> slightly less ignorant than everybody else :) and we tracked the
> behaviour in question all the way down to twisted.internet.tcp.
> The situation is this: we have an upstream data provider who sends us
> data intermittently, all day. We have downstream clients on less
> reliable connections who want to recieve it. We have a sort of caching
> proxy in the middle that keeps track of all the data sent so far
> today, and every time a client connects we send them the total record
> of the day's events, and then send them new updates as they arrive.
> The problem occurred one day when we had a higher-than-usual amount of
> data incoming. The symptom was that any client attempting to connect
> would be immediately disconnected for No Obvious Reason. After some
> ferretting about through the codebase, we discovered that the
> 'writeSomeData' method of twisted.internet.tcp.Connection was getting
> an error it couldn't handle and dropping the connection.
How many clients are we talking about? A bit of investigation leads
me to believe this can be caused by reaching the maximum number of open
sockets. Having unreliable connections can exacerbate this, since
TIME_WAIT sockets are counted towards the limit (Windows NT 4.0 seems to
leave sockets in this state for 4 minutes).
Twisted should still probably handle it, but if this is the true
cause, it should be handled in a fashion similar to errors from
accept(), possibly (a better understanding of the problem is probably
More information about the Twisted-Python