[Twisted-Python] Simpler Twisted deferred code via decorated callbacks
Terry Jones
terry at jon.es
Mon Oct 15 13:31:50 MDT 2012
Here's a description of what the 'callback' decorator in my code does.
Here's some normal Twisted code, assuming you do from twisted.web.client
import getPage (yes, getPage is kind-of obsolete, but it's a concrete and
conceptually simple deferred-returning function I like to use in examples).
def pageLength(url):
def _length(page):
return len(page)
return getPage(url).addCallback(_length)
Using the 'callback' decorator you could instead write:
def pageLength(url):
@callback
def _length(page):
return len(page)
return _length(getPage(url)
The syntactic difference in this case is minimal. The saving is more
significant in other cases, but let's consider this simplest case.
The decorator returns a wrapping function that essentially does this:
1. Look at all its arguments. For any that are Deferreds, put them into
a deferred list, which I'll call DL for this explanation. This is done
for positional args (line 25) and keyword args (line 31).
2. If no arguments are Deferreds, just call the original _length function
with the original arguments, and return the result in a Deferred using
maybeDeferred (line 44). So, any function that is called with a set
of arguments that are not Deferred instances just returns a deferred
that fires with the function result or fails if the function raises
(maybeDeferred takes care of that for us).
3. The interesting part is when some of the arguments are deferreds (line
36). In that case, we have a DeferredList called DL that will
eventually fire when all the arguments are available. The arguments
are collected, as they arrive, into a list (fags) and a dictionary
(fkw). Args that were not deferreds are already put into fargs and
fkw (lines 30 and 35).
4. If DL fires normally, we can go ahead and call the wrapped original
function with all the arguments it was supposed to receive. In our
_length case, there was only one argument, a deferred returning the
content of a web page, so the original _length gets called with the
page content. A lambda to call the original function is added to DL as
a callback (line 41), and DL is returned. Note that DL has already
fired, so adding the callback function to call the original function
results in the function being called immediately. By adding it as a
callback to DL, we can return a deferred (DL) from the wrapper.
5. When DL is made, I pass fireOnOneErrback=True to it. So if any of the
arguments to the wrapped function result in an error (i.e., in our
example, getPage fails), then DL will fail immediately. In that case,
an errback (getSubFailure) attached to DL will pull the original
failure out of DL and return it. Because one of the arguments to the
original function has failed to materialize, we obviously can't call
the original function. By just returning DL (which has failed), the
failure is propagated (in a deferred) to any caller we may have.
If you look at the code for the 'callback' decorator in decorate.py, the
above should help to make things clearer. I'm happy to answer any
questions, of course.
I hope the description makes it clear how the decorator works on a function
called with multiple deferred arguments. It's quite like a DeferredList
that you can put non-deferreds objects into and add a callback to (where
the callback is the decorated function). I implemented that once years ago,
but didn't bother to post it.
The 'errback' decorator is similar in structure, but operates differently
of course. I'll send a separate mail describing it.
Terry
More information about the Twisted-Python
mailing list