[Twisted-Python] Idiom for database access within iterator/generator?

[Itamar Shtull-Trauring]

On Tue, 2008-06-03 at 17:03 +1000, Justin Warren wrote:
> Greetings twisted gurus!
> 
> I'm hoping someone with a better algorithmic brain than I could offer
> some advice on a good way to do something in a twisty/pythonic way.
> 
> The pseudocode for what I want to do is:
> 
> for item in generator_that_fetches_rows_from_database:
> 	...
> 	do_stuff()
> 	...
> 
> where generator_that_fetches_rows_from_database is an object that uses
> enterprise.adbapi (or anything else) to fetch rows from a database and
> return them to the for..in.. loop.

runInteraction is your friend:

def getData(txn, key):
    # this runs in thread
    txn.execute("SELECT item FROM table WHERE key = :key", [key])
    return [l[o] for l in txn.fetchall()]

def gotItems(items):
    # this runs in Twisted thread
    for item in items:
        # etc...

dbpool.runInteraction(getData, 23).addCallback(gotItems)

Once you've gotten the list down to the Twisted thread you may find
twisted.internet.task.coiterate useful.