[Twisted-Python] Re: Using twisted for web crawling
gns24 at sphinx.mythic-beasts.com
Mon Jun 20 06:49:50 EDT 2005
I'm trying to write a python utility to fetch large numbers of web pages.
I originally wrote a multi-threaded app, but found that I had no control
I then re-wrote it using asyncore, which is fine apart from the fact that
a few connections open sockets which never get closed, and eventually I
run out of sockets (well, file descriptors). I see Twisted can avoid this
problem by setting an interval after which an inactive connection is closed.
I've been browsing the Twisted site and don't really understand what I
would need to implement this utility using Twisted. Is Twisted the right
technology for this, or is there a simple way to fix my 'socket leak' that
I am getting with asyncore?
Thanks in advance,
More information about the Twisted-Python