[Twisted-Python] Simpler Twisted deferred code via decorated callbacks

[Terry Jones]

Here's a description of what the 'callback' decorator in my code does.

Here's some normal Twisted code, assuming you do from twisted.web.client
import getPage (yes, getPage is kind-of obsolete, but it's a concrete and
conceptually simple deferred-returning function I like to use in examples).

    def pageLength(url):
        def _length(page):
            return len(page)
        return getPage(url).addCallback(_length)

Using the 'callback' decorator you could instead write:

    def pageLength(url):
        @callback
        def _length(page):
            return len(page)
        return _length(getPage(url)

The syntactic difference in this case is minimal. The saving is more
significant in other cases, but let's consider this simplest case.

The decorator returns a wrapping function that essentially does this:

  1. Look at all its arguments.  For any that are Deferreds, put them into
     a deferred list, which I'll call DL for this explanation. This is done
     for positional args (line 25) and keyword args (line 31).

  2. If no arguments are Deferreds, just call the original _length function
     with the original arguments, and return the result in a Deferred using
     maybeDeferred (line 44).  So, any function that is called with a set
     of arguments that are not Deferred instances just returns a deferred
     that fires with the function result or fails if the function raises
     (maybeDeferred takes care of that for us).

  3. The interesting part is when some of the arguments are deferreds (line
     36). In that case, we have a DeferredList called DL that will
     eventually fire when all the arguments are available.  The arguments
     are collected, as they arrive, into a list (fags) and a dictionary
     (fkw).  Args that were not deferreds are already put into fargs and
     fkw (lines 30 and 35).

  4. If DL fires normally, we can go ahead and call the wrapped original
     function with all the arguments it was supposed to receive.  In our
     _length case, there was only one argument, a deferred returning the
     content of a web page, so the original _length gets called with the
     page content. A lambda to call the original function is added to DL as
     a callback (line 41), and DL is returned.  Note that DL has already
     fired, so adding the callback function to call the original function
     results in the function being called immediately. By adding it as a
     callback to DL, we can return a deferred (DL) from the wrapper.

  5. When DL is made, I pass fireOnOneErrback=True to it. So if any of the
     arguments to the wrapped function result in an error (i.e., in our
     example, getPage fails), then DL will fail immediately.  In that case,
     an errback (getSubFailure) attached to DL will pull the original
     failure out of DL and return it.  Because one of the arguments to the
     original function has failed to materialize, we obviously can't call
     the original function.  By just returning DL (which has failed), the
     failure is propagated (in a deferred) to any caller we may have.

If you look at the code for the 'callback' decorator in decorate.py, the
above should help to make things clearer.  I'm happy to answer any
questions, of course.

I hope the description makes it clear how the decorator works on a function
called with multiple deferred arguments.  It's quite like a DeferredList
that you can put non-deferreds objects into and add a callback to (where
the callback is the decorated function). I implemented that once years ago,
but didn't bother to post it.

The 'errback' decorator is similar in structure, but operates differently
of course.  I'll send a separate mail describing it.

Terry