[Twisted-Python] Idiom for database access within iterator/generator?

Jean-Paul Calderone exarkun at divmod.com
Tue Jun 3 13:51:55 EDT 2008

On Tue, 03 Jun 2008 18:10:13 +0100, Phil Mayers <p.mayers at imperial.ac.uk> wrote:
>Itamar Shtull-Trauring wrote:
>>On Tue, 2008-06-03 at 17:03 +1000, Justin Warren wrote:
>>>Greetings twisted gurus!
>>>I'm hoping someone with a better algorithmic brain than I could offer
>>>some advice on a good way to do something in a twisty/pythonic way.
>>>The pseudocode for what I want to do is:
>>>for item in generator_that_fetches_rows_from_database:
>>>         ...
>>>         do_stuff()
>>>         ...
>>>where generator_that_fetches_rows_from_database is an object that uses
>>>enterprise.adbapi (or anything else) to fetch rows from a database and
>>>return them to the for..in.. loop.
>>runInteraction is your friend:
>Sure, but that's exactly what the OP didn't want; doing a bulk fetch from 
>the SQL (which yes will be non-blocking for the most part)
>I think he wanted:
>def query(txn):
>   txn.execute("...")
>   while True:
>     rs = txn.fetchone()
>     if blah:
>         continue
>     if foo:
>         transform data
>     yield data
>def sqlQuery():
>   deferred_iter = runInteraction(query)
>   for def in deferred_iter:
>     row = yield def
>     # do a thing with the row e.g. push it to an Athena page
>i.e. the function that runs inside the thread pool to be able to yield 
>values, and the function that runs in the reactor to be able to iterate 
>through them in a deferred-compatible fashion.

It should be straightforward to implement this with DeferredQueue.


More information about the Twisted-Python mailing list