[Twisted-Python] Simpler Twisted deferred code via decorated callbacks

[Naveen Michaud-Agrawal]

Hi Terry,

This is a really nice approach. Thanks for sharing! Are there any downsides
or functionality that can't be accomplished using this approach? This
combined with the generator approach to deferreds will make it easier to
reason about the code flow.

Naveen

On Sun, Oct 14, 2012 at 4:40 PM, Terry Jones <terry at jon.es> wrote:

> This morning I was thinking about deferreds and how people find them
> difficult to grasp, but how they're conceptually simple once you get it.  I
> guess most of us tell people a deferred is something to hold a result that
> hasn't arrived yet. Sometimes, though, deferreds do have a result in them
> immediately (e.g., using succeed or fail to get an already-fired deferred).
>
> I wondered if it might work to tell people to think of a deferred as really
> being the result. If that were literally true, instead of writing:
>
>     d = getPage(...)
>     d.addErrback(errcheck, args)
>     d.addCallback(cleanup, args)
>     d.addCallback(reformat, args)
>     return d
>
> We might write something like:
>
>     result1 = getPage(...)
>     result2 = errcheck(result1, args)
>     result3 = cleanup(result2, args)
>     return reformat(result3, args)
>
> And if you could write that, you could obviously instead write:
>
>     return reformat(cleanup(errcheck(getPage(...), args), args), args)
>
> If we could write Twisted code that way, I think using deferreds would be
> simpler for people unfamiliar with them.
>
> In the style we're all used to, the programmer manually adds callbacks and
> errbacks.  That's basically boilerplate. It gets worse when you then need
> to also use DeferredList, etc. It's a little confusing to read deferred
> code at first, because you need to know that the deferred result/failure is
> automatically passed as the first arg to callbacks/errbacks.  It seems to
> take a year or more for people to finally realize how the callback &
> errback chains actually interact :-) Also, I wonder how comfortable
> programmers are with code ordered innermost function first, as in the
> normal d.addCallback(inner).addCallback(outer) Twisted style, versus
> outer(inner()), as in the line above.
>
> Anyway... I realized we CAN let people use the succinct style above, by
> putting boilerplate into decorators.  I wrote two decorators, called
> (surprise!) callback and errback.  You can do this:
>
>     @errback
>     def errcheck(failure, arg):
>         ...
>
>     @callback
>     def cleanup(page, arg):
>         ...
>
>     @callback
>     def reformat(page, arg):
>         ...
>
>     reformat(cleanup(errcheck(getPage(...), arg1), arg2), arg3)
>
> The deferred callback and errback chains are hooked up automatically. You
> still get a regular deferred back as the return value.
>
> And... the "deferred" aspect of the code (or at least the need to talk
> about or explain it) has conveniently vanished.
>
> You can also do things like
>
>     func1(getDeferred1(), errcheck(func2(getDeferred2(), getDeferred3())))
>
> This gets the result of deferreds 2 & 3 and (if neither fails) passes the
> result of calling func2 on both results through to func1, which is also
> called with the result of deferred 1. You don't need to use DeferredLists,
> as the decorator makes them for you. The errcheck function wont be called
> at all unless there's an error.
>
> That's nice compared to the verbose equivalent:
>
>     d1 = DeferredList([getDeferred2(), getDeferred3()])
>     d1.addCallback(func2)
>     d1.addErrback(errcheck)
>     d2 = DeferredList([getDeferred1(), d1])
>     d2.addCallback(func1)
>
> Or the more compact but awkward:
>
>     DeferredList([getDeferred(), DeferredList([getDeferred(),
> getDeferred()]).addCallback(func2).addErrback(errcheck)]).addCallback(func1)
>
> There's lots more that could be said about this, but that's enough for now.
> The code (surely not bulletproof) and some tests are at
> https://github.com/terrycojones/twisted-callback-decorators
>
> I'll add a README sometime soon.  This is still pretty much proof of
> concept, and some it could be done slightly differently. I'm happy to
> discuss in more detail if people are interested.
>
> Terry
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20121015/82fe0f81/attachment.html>