[Twisted-Python] Re: Re: more thoughts on resumable async data flows

Clark C. Evans cce at clarkevans.com
Sun Apr 6 22:35:19 EDT 2003

On Sun, Apr 06, 2003 at 03:39:30PM +0200, Philippe Lafoucrière wrote:
| How do you fire it exactly ? Moreover, the tutorial about deferred isn't so
| clear for me (twisted newbie).

The one in CVS is a bit better, bu t it still isn't by
any means perfect.  Basically 'deferred' is just a 
mechanism for managing callback chains (and errors).
| > Then FS3 is a DeferredList taking FS1 and FS2.
| Just a note : this system may be dynamic. I mean that the user can create
| the "flow" he wants (with various datasources and other blocs).


| > As for the stuff in flow.py, right now it uses a thread
| > for each database connection.  Eventually for PostgreSQL,
| > I'd like to move over to a non-threaded driver.  Gerhard
| > was working on such a thing as I remember.  Also, it may
| > be renamed since extrepum was using "flow.py" before me.
| > 
| > Anyway, the database connection is done through a
| > QueryIterator which basically sends a set of rows
| > (via fetchmany) on to the next stage.  So, the connection
| > from DS1 to FS1 and DS2 to FS2 could be done using
| > a simple flow for each.  The problem in the flow module
| > has to do with merging the two flows.  This can be done
| > by using the 'context'.
| 3 flows  ? (1 for DS1 to FS1, 1 for DS2 to FS2, and one the merge the 2
| others) 

Yes.  In my original version, when you see 'flow' think of
a pipeline.  It was the first approximation.  I'm not totally
happy with it...

| I was thinking of using only one flow. It's weird to seperate this process
| in several flow, isn't it ?

Well, your diagram sort of implies a parallel nature, which 
wasn't part of my original requirements.  I have that need
now, so flow needs to be updated to respect this.

If you want to go over this via phone, and you are in the US
or Canada I'd be happy to drop you a line... just send me
your phone number and a good time.



More information about the Twisted-Python mailing list