[Twisted-Python] Non-blocking http client?
exarkun at divmod.com
Sat Dec 13 14:17:07 EST 2008
On Sat, 13 Dec 2008 10:08:43 -0800, Erik Wickstrom <erik at erikwickstrom.com> wrote:
>I have an application that is doing some web spidering. Right now I'm
>using urllib to retrieve the URLs, but it is painfully slow. I was
>wondering if it's feasible to swap out urllib with a twisted client
>that uses deferds so I can process urls in a more "parallel" fashion?
>I've done a bunch of Googleing, but I haven't come across anything
>that I can use as a drop in replacement. If you can point me in the
>right direction I'd really appreciate it!
There is no drop-in replacement. There is an HTTP client in Twisted,
`twisted.web.client´. There is also a more featureful HTTP client in
development but not yet available in a release of Twisted. You can
find an example of how to use the former in the Twisted Web examples,
you can follow progress of the latter at <http://twistedmatrix.com/trac/ticket/886>.
More information about the Twisted-Python