[Twisted-Python] etag and last-modified

Jacob Friis lists at debpro.webcom.dk
Fri Nov 5 09:53:34 EST 2004


exarkun at divmod.com wrote:
>>Instead I'll let Feedparser do the download and use twisted for threads.
>>What is the maximum pool size I can use?
> 
>   Using threads to do this, there is no point using Twisted at all.  Since Twisted is perfectly capable of downloading select web pages based on their headers, there's no reason to use threads.

But I need to download approx 150000 files several times every day from 
approx 7000 servers. That's why I thought threading would be the solution.
Can I use Twisted for this?

Thanks,
Jacob




More information about the Twisted-Python mailing list