[Twisted-Python] Thoughts on Deferred

Clark C. Evans cce at clarkevans.com
Thu Mar 6 10:03:25 EST 2003

On Thu, Mar 06, 2003 at 05:34:29AM -0600, Glyph Lefkowitz wrote:
| On Thursday, March 6, 2003, at 12:02 AM, Clark C. Evans wrote:
| >  1.  The entire deferred processing chain is quite nice,
| >      although the bulk of the time my callbacks don't take
| >      any arguments.  I suspect that most people who do have
| >      callback args can just make an object and pass a bound
| >      method as their callback instead of the 'args' mechanism.
| Interesting idea, but I'm not sure about it yet.  Need to think it over.

Ok.  I'll post a patch for approval.

| >  2.  The error handling could use help.
| You don't seem to understand it completely.  See itamar's post.

I think I was proposing an functionally equivalent option to
the current implementation.  After some thought, there really
isn't an advantage either way, although your implementation
could be ever-so-slightly-faster.   Let's drop this one.

| >  3.  Deferred also "artifically limits" so that the entire callback
| >      tree can only be done once.   This involves a hack 
| >"MultiDeferred"
| >      to solve the problem; but with a slightly different _runCallbacks
| >      plus a __init__ flag, this need not be the case.
| If you actually try this and figure out how to get it work with the 
| cases that Deferreds are used, I would be interested in seeing it, 
| since this is a common criticism.  However, Deferreds are not event 
| broadcasters, they are deferred results, and requests that are made 
| once should only be answered once.

Yes, this just doesn't fit since I want to provide for "partial" 
results along the way; aka a database query or a fragment of 
a file, etc.   

|  The following behavior, for example:
| 	d = Deferred()
| 	d.addCallback(foo)
| 	d.callback(bar)
| 	d.addCallback(baz)

Is this common?  

I think in a multi-result case, you can't add additional 
callbacks once 'callback' has been invoked; this restriction
is needed since you don't want to cache every value 
just-in-case another callback would be added later.

| only calls 'foo' and 'baz' once, which makes perfect sense: each 
| callback is called as soon as possible with the result at that point in 
| the chain.

Right; and any modifications would maintain the current behavior.  The
implementation path would add an additional flag to Deferred to indicate
if it was a multi-result thingy.   I'm still trying to figure out how
to send a 'finished' message down the chain.  I think this is somewhat
related to consumer/producer as well... a bit more thought is needed.

| In a "multi-"deferred case, though, what happens?  We add another 
| 'd.callback(boz)' to the end.  Then what?  does 'foo' get called once 
| and 'baz' get called once?  'foo' called twice and 'baz' called once?  
| 'foo' and 'baz' both called twice with foo and with bar?

You'd have one more call of 'foo' and 'bar', 'foo' passed 'boz', 
assuming that the callback(bar) happened after addCallback(baz).
I can't see how you'd make it work otherwise, beacuse you don't
really want to cache the value.

| This makes callback execution highly dependent upon the order in which 
| callbacks are registered.

It is already hightly dependent on this... addCallback(foo) followed
by addCallback(bar) is very different than addCallback(bar) followed
by addCallback(foo).  ;)

| Having to care about this on a regular basis smacks of the same 
| sort of mandatory awareness of order-of-execution that thread
| programming brings.

Agreed.  So you probably could'nt support the full semantics, i.e.,
the call chain would be much less dynamic.

| >  4.  Lastly, I'd like to see other "state" variables for the deferred.
| >      Essentially, what I see is a process flow, aka state transition
| >      mechnanism emerging.   For an SQL query, one of the states
| >      is 'good', 'bad', and 'finished'.  Perhaps I'm a bit off here,
| >      but being able to handle more than two states could be useful.
| This is why the Failure object has an exception type.  See the trap 
| method.  For things that aren't errors you should be able to 
| encapsulate that state information in the result.

Ok.  I was just trying to differentate between a 'chunk' notify
and a finish message.  Probably this is another difference 
between the two Deferred and MultiDeferred.  Perhaps they should
stay distinct then.

More information about the Twisted-Python mailing list