[Twisted-Python] Idiom for database access within iterator/generator?

[Jean-Paul Calderone]

On Tue, 03 Jun 2008 18:10:13 +0100, Phil Mayers <p.mayers at imperial.ac.uk> wrote:
>Itamar Shtull-Trauring wrote:
>>On Tue, 2008-06-03 at 17:03 +1000, Justin Warren wrote:
>>>Greetings twisted gurus!
>>>
>>>I'm hoping someone with a better algorithmic brain than I could offer
>>>some advice on a good way to do something in a twisty/pythonic way.
>>>
>>>The pseudocode for what I want to do is:
>>>
>>>for item in generator_that_fetches_rows_from_database:
>>>         ...
>>>         do_stuff()
>>>         ...
>>>
>>>where generator_that_fetches_rows_from_database is an object that uses
>>>enterprise.adbapi (or anything else) to fetch rows from a database and
>>>return them to the for..in.. loop.
>>
>>runInteraction is your friend:
>
>Sure, but that's exactly what the OP didn't want; doing a bulk fetch from 
>the SQL (which yes will be non-blocking for the most part)
>
>I think he wanted:
>
>def query(txn):
>   txn.execute("...")
>   while True:
>     rs = txn.fetchone()
>     if blah:
>         continue
>     if foo:
>         transform data
>     yield data
>
>
>
>@defer.inlineCallbacks
>def sqlQuery():
>   deferred_iter = runInteraction(query)
>   for def in deferred_iter:
>     row = yield def
>     # do a thing with the row e.g. push it to an Athena page
>
>i.e. the function that runs inside the thread pool to be able to yield 
>values, and the function that runs in the reactor to be able to iterate 
>through them in a deferred-compatible fashion.

It should be straightforward to implement this with DeferredQueue.

Jean-Paul