[Twisted-Python] Idiom for database access within iterator/generator?

Justin Warren daedalus at eigenmagic.com
Tue Jun 3 03:03:57 EDT 2008

Greetings twisted gurus!

I'm hoping someone with a better algorithmic brain than I could offer
some advice on a good way to do something in a twisty/pythonic way.

The pseudocode for what I want to do is:

for item in generator_that_fetches_rows_from_database:

where generator_that_fetches_rows_from_database is an object that uses
enterprise.adbapi (or anything else) to fetch rows from a database and
return them to the for..in.. loop.

Now, I'm probably looking at this from too much of a blocking code
perspective, rather than setting up a Deferred() chain of some sort, but
rather than hacking something functional but ugly, it occurs to me that
someone is bound to have done this before.

It would be nice if the database accessing object could fetch rows as
required, via cursors or whatever, rather than sucking an entire table
into memory and then doling out one row at a time... which is what I've
got working at the moment.

The morbidly curious can recoil in horror at the current code in the
SQLIterator class here:

Any free clues?

Justin Warren <daedalus at eigenmagic.com>

More information about the Twisted-Python mailing list