[Twisted-Python] Idiom for database access within iterator/generator?
exarkun at divmod.com
Tue Jun 3 13:51:55 EDT 2008
On Tue, 03 Jun 2008 18:10:13 +0100, Phil Mayers <p.mayers at imperial.ac.uk> wrote:
>Itamar Shtull-Trauring wrote:
>>On Tue, 2008-06-03 at 17:03 +1000, Justin Warren wrote:
>>>Greetings twisted gurus!
>>>I'm hoping someone with a better algorithmic brain than I could offer
>>>some advice on a good way to do something in a twisty/pythonic way.
>>>The pseudocode for what I want to do is:
>>>for item in generator_that_fetches_rows_from_database:
>>>where generator_that_fetches_rows_from_database is an object that uses
>>>enterprise.adbapi (or anything else) to fetch rows from a database and
>>>return them to the for..in.. loop.
>>runInteraction is your friend:
>Sure, but that's exactly what the OP didn't want; doing a bulk fetch from
>the SQL (which yes will be non-blocking for the most part)
>I think he wanted:
> while True:
> rs = txn.fetchone()
> if blah:
> if foo:
> transform data
> yield data
> deferred_iter = runInteraction(query)
> for def in deferred_iter:
> row = yield def
> # do a thing with the row e.g. push it to an Athena page
>i.e. the function that runs inside the thread pool to be able to yield
>values, and the function that runs in the reactor to be able to iterate
>through them in a deferred-compatible fashion.
It should be straightforward to implement this with DeferredQueue.
More information about the Twisted-Python