[Twisted-Python] Re: [flow.py] : iterator problems

Clark C. Evans cce at clarkevans.com
Sun Apr 20 13:14:35 EDT 2003

On Sun, Apr 20, 2003 at 12:28:24PM +0200, Philippe Lafoucrière wrote:
| when I yield a producer, I got in one shot the complete set of results !

Sorry about that.  There are two ways to make Threaded the threaded
wrapper operate.  In one case, the default, the wrapper yields for
every time that the wrapped iterator has returned.  In the other,
it assumes that the wrapped iterator returns a lists, and then 
yields for each item in those lists.

When I split QueryIterator and Threaded, there wasn't a way to 
let Threaded know that QueryIterator wants the second behavior,
so to do that you need to use the 'extend' option:

   flow.Threaded(flow.QueryIterator(...), extend = 1)

Anyway, I assume you are using the defaults, fetchall=0 for the 
QueryIterator, and extend = 0, for the Threaded wrapper.  In this
case, you could get N yields with M rows in them.  This is probably
the most efficient mechanism, as you spend less time in the flow
module, yet the results are incremental.   

| --------------------------------------------------------------------------
| Processed result :  ((1L, 1L), (12L, 90L), (1300L, 560L), (2000L, 10L),
| (30L, 900L), (5000L, 500L))
| --------------------------------------------------------------------------

Right, and if your result set had more rows you'd get...

| --------------------------------------------------------------------------
| Processed result :  ((1L, 1L), (12L, 90L), ... )
| Processed result :  ((1300L, 560L), (2000L, 10L), ... )
| Processed result :  ((30L, 900L), (5000L, 500L), ... )
| --------------------------------------------------------------------------

Although, to make it easier for programming, the 
extend = 1 option would make it return results...

| --------------------------------------------------------------------------
| Processed result : (1L, 1L)
| Processed result : (12L, 90L)
| Processed result : (1300L, 560L)
| Processed result : (2000L, 10L)
| Processed result : (30L, 900L)
| Processed result : (5000L, 500L)
| --------------------------------------------------------------------------

| I saw you changed many things since few days, and I'm not sure I will be
| able to correct this bug alone. By the way, I'd like to work on my
| project today, could you help me please ?

Sure.  I'll be around all day on IRC.


More information about the Twisted-Python mailing list