[Twisted-Python] understanding deferreds

[Terry Jones]

>>>>> "Alexey" == twisted-web  <twisted-web at udmvt.ru> writes:

Alexey> Is it true, that adding many callback functions as filters is
Alexey> elegant, but impractical solution?

Many callbacks as a filter is elegant & practical.

But there's a difference with errbacks, as people are touching on. That's,
with a bit of hand-waving, because when things are going right there's only
one thing happening - you're in the middle of a computation figuring out
the single result. With errors though, there are multiple types of errors,
many ways of dealing with them, multiple different possible outcomes when
an error occurs.

So the callback chain is great when everything is working, but the errback
chain can sometimes feel like it's not quite the right tool for the job.
But there are some techniques to mitigate that, and I think they work quite
well.

Alexey> Since there is only a [linear] list of pairs (callback, errback),
Alexey> the last errbacks have to be complex to distinguish the failures,
Alexey> that came from the previous callback from failures, that traverse
Alexey> the errback chain from the beginning.

Alexey> Should I only use errbacks as a means of catching errors only from
Alexey> who have produced the Deferred?  And never use them to catch
Alexey> errors, that came from my callbacks?

No. One thing to think about is the flow down the call/errback chain. You
can do something like this.

  d = IReturnADeferred()

  d.addErrback(e1)
  d.addCallback(c1)

  d.addErrback(e2, 'first')

  d.addCallback(c2)
  d.addCallback(c3)
  d.addErrback(e2, 'second')

Here e1 will only get errors created by IReturnADeferred. If
IReturnADeferred has no problem, you're into c1 and e1 is "behind" you on
the call/errback chain. Note that I add e2 to the errback chain twice.
That's a way of having the same function be called for (perhaps) the same
reason, but on the chain in 2 places. By the time the flow of control gets
to c2 or c3, e2 can only be called with 'second' as its argument.

Alexey> Nevertheless the Deferred mechanism always catch the exceptions in
Alexey> my callbacks, I have to always catch them myself and never let them
Alexey> out uncontrolled (since I cannot distinguish, say TypeError that
Alexey> came from the first callback from TypeError, that came from the
Alexey> next callback, and they are not the same for me, since I try to do
Alexey> a cleanup)?  But then I have to translate all errors to my invented
Alexey> exceptions, so they will differ.

See the above example.

Alexey> Or just tell me, that I have missed something, I'm still getting
Alexey> into the theme!

The other thing that's very useful is to write errbacks like this:

  def e1(fail):
    fail.trap(TypeError, KeyError)
    # Do stuff to handle just TypeError, KeyError

  def e2(fail):
    fail.trap(MyException)
    # Do stuff to handle MyException

then you just do

  d = IReturnADeferred()
  d.addErrback(e1)
  d.addErrback(e2)

The fail.trap will cause the function to immediately return the current
failure if it's not one of the mentioned types. See also fail.check (which
is what trap uses).  I.e., go read the source of twisted.python.failure

That approach is nice because you can write your err handling methods
completely separately, and can probably not worry about the order you list
them in, etc. The failure just falls down the errback chain until it hits
something that can deal with it ("trap" it).


Another thing you could do (a bit of a hack) is to add an attribute to a
failure object to mark that you've processed it (e.g., that it has been
logged).

And you might want to read the recent post to this list by Brian Warner
(which I've also been meaning to reply to - hi Brian!). See
http://twistedmatrix.com/pipermail/twisted-python/2010-January/021330.html

Probably others have suggestions on nice ways to handle particular
cases. Those are just some that I've used.

Terry