[Twisted-web] Proxy Could not accept new connection

[Glyph Lefkowitz]

On Thu, Aug 6, 2009 at 6:22 PM, dave fowler <davefowler at gmail.com> wrote:

> I guys.  I'm running a twisted web proxy and after a day or so of usage it
> stops working and start spitting out these errors.
>
> 2009/08/05 14:04 +0000 [__main__.ProxyFactory] Could not accept new
> connection (EMFILE)
> 2009/08/05 14:04 +0000 [__main__.ProxyFactory] Could not accept new
> connection (EMFILE)
> 2009/08/05 14:04 +0000 [__main__.ProxyFactory] Could not accept new
> connection (EMFILE)
> 2009/08/05 14:04 +0000 [__main__.ProxyFactory] Could not accept new
> connection (EMFILE)
> 2009/08/05 14:04 +0000 [__main__.ProxyFactory] Could not accept new
> connection (EMFILE)
> 2009/08/05 14:04 +0000 [__main__.ProxyFactory] Could not accept new
> connection (EMFILE)
>
> Can anyone expand on what this error is or suggestions on fixing it or
> debugging further?
>
> What is the EMFILE?
>

It means you're out of file descriptors for your process.

See here for the root cause: <http://linux.die.net/man/2/accept>,
specifically the part about EMFILE in the Errors section.  I'm actually glad
to see this error report - Twisted used to fail in a *much* worse way in
this error condition (<http://twistedmatrix.com/trac/ticket/662>), this is
the first proof I've seen that our fix works in the wild :).

To increase your per-process file-descriptor limit, use "ulimit -n".  Be
aware that in order to increase this above 1024, you'll need to be root; I
think there's a way to set it up to be bigger than that for non-root users
but I don't know how.

In order to free up file descriptors, keep a list of your open connections
and shut down some of them.  Sadly, your Factory object won't be notified of
this problem, so you might have to hack something gross together (like a log
observer that watches for the string "(EMFILE)").  I just filed this ticket
to provide a nice, clean notification: <
http://twistedmatrix.com/trac/ticket/3958>.  This may be a duplicate of
something, but if so it should be closed with a link to what it's a
duplicate of.

In some cases, connections won't shut down properly because there is still
some buffered data and the other end is not available to acknowledge the
connection's closure.  If this is the case, you can forcibly close the file
descriptor (although this may cause some ugliness).  To get a good,
supported way to destroy file descriptors in Twisted, see this ticket: <
http://twistedmatrix.com/trac/ticket/78>.

Good luck!  Sorry Twisted has not handled this error in the best way for
you, I hope you help us fix some of these bugs for the next guy :).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-web/attachments/20090806/29cbefc1/attachment.htm