[Twisted-Python] ludicrous timeouts in protocols.http.HTTPFactory and web.server.Site

Antoine Pitrou solipsis at pitrou.net
Thu Dec 30 07:44:32 MST 2004


Hi,

I'm accessing a twisted-based XMLRPC server through an HTTP proxy which
seems to leave outgoing connections open for a very long time (I think
it waits for the server to close them on its own). This means that after
a handful of remote method calls through this proxy, my server runs into
file descriptor limits:

Traceback (most recent call last):
  File "/usr/lib/python2.3/site-packages/twisted/python/log.py", line 65, in callWithLogger
  File "/usr/lib/python2.3/site-packages/twisted/python/log.py", line 52, in callWithContext
  File "/usr/lib/python2.3/site-packages/twisted/python/context.py", line 43, in callWithContext
  File "/usr/lib/python2.3/site-packages/twisted/internet/default.py", line 535, in _doReadOrWrite
--- <exception caught here> ---
  File "/usr/lib/python2.3/site-packages/twisted/internet/tcp.py", line 625, in doRead
  File "/usr/src/build/475206-i386/install/usr/lib/python2.3/socket.py", line 167, in accept
socket.error: (24, 'Too many open files')

Which is easily checked by a netstat dump :

$ netstat --inet --tcp -n -p | grep :8550
tcp        0      0 192.33.178.29:8550          193.49.124.107:30839        ESTABLISHED 3076/python 
tcp        0      0 192.33.178.29:8550          193.49.124.107:25463        ESTABLISHED 3076/python 
tcp        0      0 192.33.178.29:8550          193.49.124.107:37494        ESTABLISHED 3076/python 
tcp        0      0 192.33.178.29:8550          193.49.124.107:33910        ESTABLISHED 3076/python 
tcp        0      0 192.33.178.29:8550          193.49.124.107:20854        ESTABLISHED 3076/python 
...
... and so on for exactly 998 lines


Looking at the Twisted code, I see that very long timeouts have been
defined for both protocols.http.HTTPFactory and web.server.Site
(60*60*12, that is 12 hours!). If I override the "timeout" parameter
when constructing the Site receiving XMLRPC connections, then the
problem disappears:
        reactor.listenTCP(xmlrpc_port, server.Site(self, timeout=30), interface=xmlrpc_host)

I think the default values in Twisted are quite bogus and should be
changed to more sensible ones. 30 or 60 seconds is ok in the context of
an HTTP connection. Very long timeouts on the other hand make the server
very vulnerable.

Regards

Antoine.






More information about the Twisted-Python mailing list