[Twisted-Python] Re: Using twisted for web crawling

Graham Stratton gns24 at sphinx.mythic-beasts.com
Mon Jun 20 06:49:50 EDT 2005


Hello all!

I'm trying to write a python utility to fetch large numbers of web pages.
I originally wrote a multi-threaded app, but found that I had no control
of it.

I then re-wrote it using asyncore, which is fine apart from the fact that
a few connections open sockets which never get closed, and eventually I
run out of sockets (well, file descriptors).  I see Twisted can avoid this
problem by setting an interval after which an inactive connection is closed.

I've been browsing the Twisted site and don't really understand what I
would need to implement this utility using Twisted.  Is Twisted the right
technology for this, or is there a simple way to fix my 'socket leak' that
I am getting with asyncore?

Thanks in advance,

Graham





More information about the Twisted-Python mailing list