[Twisted-Python] deferred generators in inline callbacks
exarkun at twistedmatrix.com
exarkun at twistedmatrix.com
Tue Dec 20 14:14:10 EST 2011
On 06:24 pm, alex.kirp at gmail.com wrote:
>Hi.
>
>I have a question about inline callbacks usage.
>
>Let's assume I have an iterator that works synchronously. I want to
>wrap this generator to work asynchronously in threads:
>
>def some_generator():
> cursor = make_iterator() # cursor's 'next' method uses some nasty
>blocking I/O
> while 1:
> yield deferToThread(cursor.next)
>
>
>Next, I want to asynchronously iterate over this generator in
>inlineCallbacks semantics. I should say:
>
>@inlineCallbacks
>def do_stuff():
> for item in some_generator():
> try:
> real_item = yield item
> # do stuff with real_item
> except StopIteration:
> break
>
>
>It somewhat annoys me that I should always convert item to real_item
>so inlineCallbacks can properly asynchronize my code. I also need to
>implement StopIteration myself :(
>Is there any easier way to do this in Twisted?
You're basically looking for coroutines - ie, context switching across
multiple stack frames (from inside some_generator through do_stuff out
to the implementation of inlineCallbacks).
There are several coroutine libraries for Python. Greenlets is probably
the most popular. You can use it with Twisted, if you want.
Personally, I'd probably go for something more like:
def worker(item):
# do stuff with item
hook_up(some_generator(), worker)
with some kind of hook_up that does the parts of this task that you're
tired of repeating over and over again (like turning item into
real_item).
The result doesn't require figuring out how your coroutines are
affecting control flow and probably gives you something more composable
too.
Jean-Paul
More information about the Twisted-Python
mailing list