[Twisted-Python] Thoughts on Deferred

[Clark C. Evans]

Ok.  I've been using Deferreds some time now, actually I've
coverted my report server to now use the mechanism as my 
core 'construction' model.   I'd like to report some 
observations:

  1.  The entire deferred processing chain is quite nice, 
      although the bulk of the time my callbacks don't take
      any arguments.  I suspect that most people who do have
      callback args can just make an object and pass a bound
      method as their callback instead of the 'args' mechanism.
      This mechanism could be better supported by providing 
      a wrapper class instead of maintaining all of the args 
      throught the code:

          class DispatchCallback:
              def __init__(self, callback, args, kw):
                  self.callback = callback
                  self.args = args or ()
                  self.kw   = kw or {}
              def apply(self,result):
                  return apply(self.callback, (result,)+self.args, kw)

      Then, in addCallbacks, if any callback was given which had
      args or a kw, you could just construct this dispatch object
      and put the ob.apply into the callback stack.   In this 
      way only users who passed in callbacks with args pay for the
      penalty of applying them.

  2.  The error handling could use help.  The cross-over behavior,
      while unique, doesn't provide any more value over a simple 
      stack /w state flag (good/bad) attach to each callback.

        def _addCallback(self, callback, *args=None, **kw=None, state=good):
            if args or kw: callback = DispatchCallback(callback,args,kw)
            self.callbacks.append((state,callback))

      Of course, existing functions can then be expressed as a
      sequence of addCallback /w state flag.
            
            addCallback  => _addCallback(self,callback,args,kw,state=good)
            addErrback   => _addCallback(self,errback,args,kw,state=bad)
            addCallbacks => addErrback(...); addCallback(...);
           
      It would have the same behavior since the current cross/over
      can easily be denormalized into two entries into the stack, one
      for a bad state (first) and then one for a good state (second).
      With these changes in place, _runCallbacks() is much simpler...

  3.  Deferred also "artifically limits" so that the entire callback
      tree can only be done once.   This involves a hack "MultiDeferred"
      to solve the problem; but with a slightly different _runCallbacks
      plus a __init__ flag, this need not be the case.

      In short, the "multi-call deferred" that I need shouldn't be
      a separate class, its behavior can be rolled into the core Deferred
      without changing existing behavior (default to callOnceOnly).

  4.  Lastly, I'd like to see other "state" variables for the deferred. 
      Essentially, what I see is a process flow, aka state transition
      mechnanism emerging.   For an SQL query, one of the states
      is 'good', 'bad', and 'finished'.  Perhaps I'm a bit off here,
      but being able to handle more than two states could be useful.
      
      I say this beacuse the MultiDeferred class has a addFinishCallback,
      which is fired on the 'finish' state.  Note that errors in the 
      'row' callback and the 'finish' callback could possibly be treated
      the same, so there probably isn't a reason to split this... but
      maybye not.. I'm still thinking this one out.

Just some thoughts.  If you like, I could refactor the Deferred
object as above (it'd be less code & probably cleaner), as well
as updating the documentation and providing examples which run
without modification.

All in all, this is a great concept... and it's working beautyfully
in my current application (mod the minor blemishes above).

Best,

Clark