[Twisted-Python] Non-blocking http client?
erik at erikwickstrom.com
Sat Dec 13 13:08:43 EST 2008
I have an application that is doing some web spidering. Right now I'm
using urllib to retrieve the URLs, but it is painfully slow. I was
wondering if it's feasible to swap out urllib with a twisted client
that uses deferds so I can process urls in a more "parallel" fashion?
I've done a bunch of Googleing, but I haven't come across anything
that I can use as a drop in replacement. If you can point me in the
right direction I'd really appreciate it!
Thanks for your help!
More information about the Twisted-Python