[Twisted-Python] Simpler Twisted deferred code via decorated callbacks

[Terry Jones]

This morning I was thinking about deferreds and how people find them
difficult to grasp, but how they're conceptually simple once you get it.  I
guess most of us tell people a deferred is something to hold a result that
hasn't arrived yet. Sometimes, though, deferreds do have a result in them
immediately (e.g., using succeed or fail to get an already-fired deferred).

I wondered if it might work to tell people to think of a deferred as really
being the result. If that were literally true, instead of writing:

    d = getPage(...)
    d.addErrback(errcheck, args)
    d.addCallback(cleanup, args)
    d.addCallback(reformat, args)
    return d

We might write something like:

    result1 = getPage(...)
    result2 = errcheck(result1, args)
    result3 = cleanup(result2, args)
    return reformat(result3, args)

And if you could write that, you could obviously instead write:

    return reformat(cleanup(errcheck(getPage(...), args), args), args)

If we could write Twisted code that way, I think using deferreds would be
simpler for people unfamiliar with them.

In the style we're all used to, the programmer manually adds callbacks and
errbacks.  That's basically boilerplate. It gets worse when you then need
to also use DeferredList, etc. It's a little confusing to read deferred
code at first, because you need to know that the deferred result/failure is
automatically passed as the first arg to callbacks/errbacks.  It seems to
take a year or more for people to finally realize how the callback &
errback chains actually interact :-) Also, I wonder how comfortable
programmers are with code ordered innermost function first, as in the
normal d.addCallback(inner).addCallback(outer) Twisted style, versus
outer(inner()), as in the line above.

Anyway... I realized we CAN let people use the succinct style above, by
putting boilerplate into decorators.  I wrote two decorators, called
(surprise!) callback and errback.  You can do this:

    @errback
    def errcheck(failure, arg):
        ...

    @callback
    def cleanup(page, arg):
        ...

    @callback
    def reformat(page, arg):
        ...

    reformat(cleanup(errcheck(getPage(...), arg1), arg2), arg3)

The deferred callback and errback chains are hooked up automatically. You
still get a regular deferred back as the return value.

And... the "deferred" aspect of the code (or at least the need to talk
about or explain it) has conveniently vanished.

You can also do things like

    func1(getDeferred1(), errcheck(func2(getDeferred2(), getDeferred3())))

This gets the result of deferreds 2 & 3 and (if neither fails) passes the
result of calling func2 on both results through to func1, which is also
called with the result of deferred 1. You don't need to use DeferredLists,
as the decorator makes them for you. The errcheck function wont be called
at all unless there's an error.

That's nice compared to the verbose equivalent:

    d1 = DeferredList([getDeferred2(), getDeferred3()])
    d1.addCallback(func2)
    d1.addErrback(errcheck)
    d2 = DeferredList([getDeferred1(), d1])
    d2.addCallback(func1)

Or the more compact but awkward:

    DeferredList([getDeferred(), DeferredList([getDeferred(), getDeferred()]).addCallback(func2).addErrback(errcheck)]).addCallback(func1)

There's lots more that could be said about this, but that's enough for now.
The code (surely not bulletproof) and some tests are at
https://github.com/terrycojones/twisted-callback-decorators

I'll add a README sometime soon.  This is still pretty much proof of
concept, and some it could be done slightly differently. I'm happy to
discuss in more detail if people are interested.

Terry