[Twisted-Python] Idiom for database access within iterator/generator?

Phil Mayers p.mayers at imperial.ac.uk
Tue Jun 3 13:10:13 EDT 2008


Itamar Shtull-Trauring wrote:
> On Tue, 2008-06-03 at 17:03 +1000, Justin Warren wrote:
>> Greetings twisted gurus!
>>
>> I'm hoping someone with a better algorithmic brain than I could offer
>> some advice on a good way to do something in a twisty/pythonic way.
>>
>> The pseudocode for what I want to do is:
>>
>> for item in generator_that_fetches_rows_from_database:
>> 	...
>> 	do_stuff()
>> 	...
>>
>> where generator_that_fetches_rows_from_database is an object that uses
>> enterprise.adbapi (or anything else) to fetch rows from a database and
>> return them to the for..in.. loop.
> 
> runInteraction is your friend:

Sure, but that's exactly what the OP didn't want; doing a bulk fetch 
from the SQL (which yes will be non-blocking for the most part)

I think he wanted:

def query(txn):
   txn.execute("...")
   while True:
     rs = txn.fetchone()
     if blah:
         continue
     if foo:
         transform data
     yield data



@defer.inlineCallbacks
def sqlQuery():
   deferred_iter = runInteraction(query)
   for def in deferred_iter:
     row = yield def
     # do a thing with the row e.g. push it to an Athena page

i.e. the function that runs inside the thread pool to be able to yield 
values, and the function that runs in the reactor to be able to iterate 
through them in a deferred-compatible fashion.

AFAIK there is no such thing, which is a shame because for *really* huge 
queries it has the potential to significantly reduce memory usage.

On a not-very related issue, it would be highly useful IMHO for





More information about the Twisted-Python mailing list