[Twisted-Python] Non-blocking http client?
denis.bilenko at gmail.com
Sun Dec 14 12:58:58 EST 2008
I've been working on eventlet-twisted integration and it includes a drop-in
replacements for standard Python modules (including urllib and httplib) that
do not block the reactor.
There was no release yet and there're no documentation written.
(You can find the documentation for the original eventlet here
however, not all of it is applicable. You can also read the
documentation of greenlet here
However, usage is very simple.
1. add this line before using any eventlet functions
from eventlet.twistedutil import join_reactor
2. import eventlet.green.urllib instead of urllib
3. Use eventlet.twistedutil.deferToGreenThread function to make
from eventlet.green.urllib return Deferred.
Here's an example on how to use eventlet.green.httplib and twisted's
together to build a toy http proxy:
To get the repository:
hg clone http://devel.ag-projects.com/~denis/cgi-bin/hgweb.cgi/eventlet_twisted
The only dependency is py.magic.greenlet from py library.
The code is only tested on python2.5 linux & macos so it probably won't work
on python2.4/windows but that will be fixed sometime in the future.
On Sun, Dec 14, 2008 at 12:08 AM, Erik Wickstrom <erik at erikwickstrom.com> wrote:
> Hi all,
> I have an application that is doing some web spidering. Right now I'm
> using urllib to retrieve the URLs, but it is painfully slow. I was
> wondering if it's feasible to swap out urllib with a twisted client
> that uses deferds so I can process urls in a more "parallel" fashion?
> I've done a bunch of Googleing, but I haven't come across anything
> that I can use as a drop in replacement. If you can point me in the
> right direction I'd really appreciate it!
> Thanks for your help!
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
More information about the Twisted-Python