[Twisted-Python] deferred generators in inline callbacks

exarkun at twistedmatrix.com exarkun at twistedmatrix.com
Tue Dec 20 14:14:10 EST 2011


On 06:24 pm, alex.kirp at gmail.com wrote:
>Hi.
>
>I have a question about inline callbacks usage.
>
>Let's assume I have an iterator that works synchronously. I want to
>wrap this generator to work asynchronously in threads:
>
>def some_generator():
>    cursor = make_iterator() # cursor's 'next' method uses some nasty
>blocking I/O
>    while 1:
>        yield deferToThread(cursor.next)
>
>
>Next, I want to asynchronously iterate over this generator in
>inlineCallbacks semantics. I should say:
>
>@inlineCallbacks
>def do_stuff():
>    for item in some_generator():
>        try:
>            real_item = yield item
>            # do stuff with real_item
>        except StopIteration:
>            break
>
>
>It somewhat annoys me that I should always convert item to real_item
>so inlineCallbacks can properly asynchronize my code. I also need to
>implement StopIteration myself :(
>Is there any easier way to do this in Twisted?

You're basically looking for coroutines - ie, context switching across 
multiple stack frames (from inside some_generator through do_stuff out 
to the implementation of inlineCallbacks).

There are several coroutine libraries for Python.  Greenlets is probably 
the most popular.  You can use it with Twisted, if you want.

Personally, I'd probably go for something more like:

    def worker(item):
        # do stuff with item

    hook_up(some_generator(), worker)

with some kind of hook_up that does the parts of this task that you're 
tired of repeating over and over again (like turning item into 
real_item).

The result doesn't require figuring out how your coroutines are 
affecting control flow and probably gives you something more composable 
too.

Jean-Paul



More information about the Twisted-Python mailing list