[Twisted-Python] Idiom for database access within iterator/generator?
p.mayers at imperial.ac.uk
Tue Jun 3 13:10:13 EDT 2008
Itamar Shtull-Trauring wrote:
> On Tue, 2008-06-03 at 17:03 +1000, Justin Warren wrote:
>> Greetings twisted gurus!
>> I'm hoping someone with a better algorithmic brain than I could offer
>> some advice on a good way to do something in a twisty/pythonic way.
>> The pseudocode for what I want to do is:
>> for item in generator_that_fetches_rows_from_database:
>> where generator_that_fetches_rows_from_database is an object that uses
>> enterprise.adbapi (or anything else) to fetch rows from a database and
>> return them to the for..in.. loop.
> runInteraction is your friend:
Sure, but that's exactly what the OP didn't want; doing a bulk fetch
from the SQL (which yes will be non-blocking for the most part)
I think he wanted:
rs = txn.fetchone()
deferred_iter = runInteraction(query)
for def in deferred_iter:
row = yield def
# do a thing with the row e.g. push it to an Athena page
i.e. the function that runs inside the thread pool to be able to yield
values, and the function that runs in the reactor to be able to iterate
through them in a deferred-compatible fashion.
AFAIK there is no such thing, which is a shame because for *really* huge
queries it has the potential to significantly reduce memory usage.
On a not-very related issue, it would be highly useful IMHO for
More information about the Twisted-Python