[Twisted-Python] Simpler Twisted deferred code via decorated callbacks

[Terry Jones]

Here's a description of what the 'errback' decorator in my code does. (BTW,
I've just updated it.)

Here's normal Twisted errback code, assuming you do from twisted.web.client
import getPage and from twisted.python import log.

    def logGetPageError(url):
        def handleError(failure, url):
            log.msg('Could not fetch URL %s.' % url)
        return getPage(url).addErrback(handleError, url)

Using the 'errback' decorator you could instead write:

    def logGetPageError(url):
        @errback
        def handleError(failure, url):
            log.msg('Could not fetch URL %s.' % url)
        return handleError(getPage(url), url)

As with the 'callback' decorator, the syntactic difference between the two
in this case is minimal.

The decorator returns a wrapping function that essentially does this:

  0. Return a failed Deferred if the decorated function is called with
     no positional arguments (line 69). The reason for this is below.

  1. Look at all its arguments.  For any that are Deferreds, put them into
     a deferred list, which I'll call DL for this explanation. This is done
     for positional args (line 76) and keyword args (line 82).  In our
     example, DL will contain the Deferred returned by getPage.

  2. Suppose none of the arguments were Deferreds. If any of the args were
     instances of Failure (line 96), return a deferred that will fire with
     the result of calling the original wrapped (errback) function with all
     the passed arguments (at least one of which is a Failure).  If no
     argument was a Failure, the original wrapped function is NOT called
     because there hasn't been any error. Instead (line 99), an
     already-fired Deferred (obtained via succeed) is returned with the
     value of the first positional argument.  This is how the wrapping
     errback function passes the result along to its caller (if any). The
     other arguments to the wrapped function were obviously intended for
     that function, and are unused (this mirrors the way args passed to
     addErrback are unused if the errback is never run due to no error
     occurring).

  3. If, as in our example, some of the arguments are deferreds (line 87)
     we have a DeferredList called DL that will eventually fire when all
     the Deferreds have fired (i.e., when all arguments are available for
     the wrapped function). Note that due to the use of addBoth (lines 79
     and 84) non-Failure and Failure arguments coming from arguments that
     were Deferreds are both collected. These are collected, as they
     arrive, into a list (fargs) and a dictionary (fkw).  Args that were
     not deferreds are already put into fargs and fkw (lines 78 and 86).

  4. If DL fires with no errors (in our case, that means getPage returns a
     page), we act as we did in step 2: If any of the args are instances of
     Failure (line 89), the 'finish' callback added to DL (line 93) will
     return the result of calling the original wrapped (errback) function
     (handleError in our case) with all the passed arguments.  If no
     argument was a Failure, the original wrapped function is NOT called
     because there hasn't been any error, and the callback on DL returns
     the first positional arg (on line 92).  This last case corresponds to
     getPage returning successfully and the handleError errback not being
     called.

The 'errback' decorator differs from the 'callback' decorator in that its
wrapper function:

  - Collects all Failures in all arguments (including any coming from
    Deferreds, of course) in order to call the wrapped errback function.
    The 'callback' decorator OTOH immediately returns a failed Deferred as
    soon as any error happens (which is why its DeferredList uses
    fireOnOneErrback=True).

  - Must be called with at least one positional argument, so that in the
    case where nothing goes wrong with any Deferred argument (and no other
    arg is already a Failure) it has a value to pass back to its caller (if
    any).

Both the wrapper functions sometimes do not call the wrapped function. This
is symmetric and is as you'd expect: the 'callback' decorator doesn't call
the wrapped (callback) function if there's any error, whereas the 'errback'
decorator doesn't call the wrapped (errback) function unless there's an
error.  This is very similar to regular Twisted call/errback processing: if
you're on the callback chain, errback functions aren't called & vice versa.
Just as in regular Twisted code, a callback can produce a Failure or an
errback can produce a non-Failure and you're back on the other chain.

If you've read all this, great :-) It should be clear that you can roll up
much more interesting cases, combining decorated callbacks and errbacks
that accept multiple Deferred and non-Deferred arguments.  Those are the
cases where the syntactic / code complexity savings are much more apparent
because (at least in many cases) there's no need to write code with
addCallback, addErrback, or to use DeferredList.

Some people's immediate reaction to all this is to tell me about
inlineCallbacks.  But this is very different, even though both share the
universal decorator goal of making code look simpler / easier.
inlineCallbacks lets you explicitly yield values from Deferreds.
Unfortunately, anything can happen while you're context switched out before
the yield yields.  The decorators I've posted just wire up regular Twisted
callback/errback processing chains, with no use of yield. There are other
differences, but this post is already way long...

Terry