[Twisted-Python] Re: Re: more thoughts on resumable async data flows
lafou at wanadoo.fr
Sun Apr 6 09:39:30 EDT 2003
> If your queries are returning small results, ie,
> non-incremental, I would just set this up as a
> chain of Deferreds and not bother with flow as
> it might be overkill for your needs.
No, this may be used with a very large amount of data.
> In other words, in the twisted.enterprise.adbapi
> when you run a query, it returns a deferred. So
> DS1 and DS2 would be those query results.
I agree with that
> Your FS1 and FS2 should be deferred operations
> as well, but fired via the 'addCallback' on
> DS1 and DS2 respectively.
How do you fire it exactly ? Moreover, the tutorial about deferred isn't so
clear for me (twisted newbie).
> Then FS3 is a DeferredList taking FS1 and FS2.
Just a note : this system may be dynamic. I mean that the user can create
the "flow" he wants (with various datasources and other blocs).
> If you can use generators, for more complicated flows,
> etrepum's (Bob Ippolito's) approach may be better approach
> to this, although I'm not sure how it would handle more
> 'fluid' flow through the process. I need to understand
> what he's doing better. At first I thought his approach
> is very different than mine, but I think (although I'm
> not quite yet able to explain) that they are isomorphic;
> only that I'm using a temp stack and he's using the
> event queue via a linked-list.
I don't really know generators. I'm going to take a look at like tomorow...
> As for the stuff in flow.py, right now it uses a thread
> for each database connection. Eventually for PostgreSQL,
> I'd like to move over to a non-threaded driver. Gerhard
> was working on such a thing as I remember. Also, it may
> be renamed since extrepum was using "flow.py" before me.
> Anyway, the database connection is done through a
> QueryIterator which basically sends a set of rows
> (via fetchmany) on to the next stage. So, the connection
> from DS1 to FS1 and DS2 to FS2 could be done using
> a simple flow for each. The problem in the flow module
> has to do with merging the two flows. This can be done
> by using the 'context'.
3 flows ? (1 for DS1 to FS1, 1 for DS2 to FS2, and one the merge the 2
I was thinking of using only one flow. It's weird to seperate this process
in several flow, isn't it ?
> In short, it would work, but I'm not sure that it's
> your best option; and I'm still not quite happy with
> flow.py myself (although it is in production).
thank you for your help
Thank very much Clark !
More information about the Twisted-Python