[Twisted-Python] If the errbacks of a canceled Deferred are called with error other than CancelledError, is this acceptable?

[Terry Jones]

First off, +1 on propagating the CancelledError failure (or something even
more specific) all the way back up the errback chain.

lvh> Personally, I think it's enough of a change in functionality to
warrant a chance in ways a function can fail

I'm not sure what change in functionality you mean. Deferreds are already
cancelable, and functions aren't going to fail in new ways. The only thing
that's new here is that a different value might be given to the
failure.Failure constructor and passed along the errback chain.

glyph> In fact in this case you almost want *multiple* inheritance, so you
can say 'except CancelledError:' or 'except ConnectionError:' as
appropriate.

If a deferred has been canceled and the errback fired, why can't code like
`fail.check(CancelledError, ConnectionError, ...)` be used?

Sorry if I'm wrong/forgetful (I'm old), but it seems like lvh & glyph are
talking about exceptions when they should be talking about handling Failure
instances arriving via the errback chain.

A couple of comments:

 - There are conceptually 3 places a deferred might be cancelled. 1. The
code that makes the deferred might call cancel for some reason (e.g.,
service shutdown). 2. Intermediate code that called the code that creates
the deferred and which passes the deferred on might cancel it. 3. The
originating code (that uses a Twisted (or other library) API call) might
decide to cancel it (e.g., due to timeout).

In case 1, the documented interface of the API call can say what it does if
it cancels a deferred. E.g., that a CancelledError failure will be
delivered down the errback chain (the default behavior), or that some other
kind of failure will be sent. A function could even allow the calling code
to pass in a value that should be sent down the errback chain (wrapped in a
Failure) when/if the deferred it creates is cancelled.  E.g. (pseudo-code):

    def doSomething(cancelValue=None):
        if cancelValue is not None:
            deferred = Deferred(lambda d:
d.errback(failure.Failure(cancelValue)))
        else:
            deferred = Deferred()
        # Now do other things to arrange for 'deferred' to fire or fail.
        return deferred

Case 2 is almost never going to happen. The intermediate code gets a
deferred from someplace, maybe adds call/errbacks to it, and passes it on.
If that code wants to hold on to the deferred it received and cancel it for
some reason, it could do so, but it wont have any control over the failure
value that comes down the errback chain.

Case 3 is more interesting, and is the main reason deferred cancelation was
interesting me to (please see this thread for some background on how we got
here:
http://twistedmatrix.com/pipermail/twisted-python/2010-January/021298.html).
In this case, the original calling code wants to cancel the deferred.
E.g., it has made a call to something that makes a network call and after
some timeout decides it needs to proceed without the result. Due to its
setup (using deferreds) the cleanest way for that code to proceed is to
trigger the deferred itself. If it can do that, the normal (errback) error
processing chain can simply handle the case where the deferred is cancelled.

A slight difficulty with the current situation is that code that obtains a
deferred made by other code can't tell, if the deferred is cancelled, who
cancelled it (or why). That includes the case where the code that received
the deferred cancels it itself. I.e., if the code that makes the deferred
cancels it (in the default way) or the code that receives the deferred
cancels it (cases 1 and 3 above), the result is the same, a CancelledError
in a failure.

In what I originally proposed (see above link), the caller could errback
the deferred it was given *with a value of its choosing*. That would allow
code to cancel a deferred and also to detect (in an errback) that it had
done so and/or why (code might cancel a deferred for different reasons).
 You can't do that with the implementation that actually landed in Twisted,
though.


A possible way to add some functionality / flexibility (including the above
possibility) in a backwards-compatible way would be to allow
Deferred.cancel to be called with a value argument (default=None to keep it
backwards compat).  If no canceller was given to the Deferred constructor
(or the canceller function did not fire the deferred), Deferred.cancel
would call self.errback(failure.Failure(CancelledError(value))). In that
way, anyone could cancel a deferred and the specific reason passed to
cancel(), if any, would be available in the Python CancelledError instance
as its first argument, as with any Python exception (i.e., args[0]).  In
addition (backwards incompatible, though) the value passed to
Deferred.cancel could also be passed to self._canceller, along with the
deferred itself. I don't think that's needed though, as code doing more
elaborate cancellation can set up any kind of Failure it wants to propagate
back, can document its behavior, and can also allow for caller-specific
values to be passed back (see code fragment above), etc.

OK, sorry if that's all a bit rambling & hard to follow...

Terry



On Tue, Jun 18, 2013 at 8:54 PM, Glyph <glyph at twistedmatrix.com> wrote:

>
> On Jun 18, 2013, at 12:03 PM, Laurens Van Houtven <_ at lvh.io> wrote:
>
> On Tue, Jun 18, 2013 at 8:22 PM, Glyph <glyph at twistedmatrix.com> wrote:
>
>> I would say that if we want to percolate this information up to the
>> caller, there should be a ConnectingCancelled exception that is a subtype
>> of the previous exception type.
>>
>
> Doesn't that mean we'll have many subclasses that mean that something was
> cancelled?
>
> If I didn't take backwards compatibility into account, I would say that
> composing the original exception into a new CancellationError (or
> something) exception would be preferable. Would you agree that it would be
> preferable? (Again, not taking compatibility into account -- I'm trying to
> get compatibility vs niceness of API to face off against each other.
> Personally, I think it's enough of a change in functionality to warrant a
> chance in ways a function can fail, but there's no point in even having
> that argument if there's no consensus that the composed way would even be
> better...)
>
>
> I agree that it would be preferable, but I don't see how it's possible
> without making Exception itself composeable.
>
>  After all, if it's interesting that the operation was cancelled,
> presumably it's interesting *at what stage* the operation is cancelled.
>
> IIUC that would work the same with composition as inheritance :)
>
>
> Unfortunately inheritance is built into the way Python handles exceptions.
>  In fact in this case you almost want *multiple* inheritance, so you can
> say 'except CancelledError:' or 'except ConnectionError:' as appropriate.
>  :-(
>
> The one saving grace here is that not a whole lot of useful logic can live
> on the exception objects, so there's a limited amount of opportunity for
> getting oneself into trouble.
>
> Please prove me wrong, though.
>
> -glyph
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20130618/e93c9bfe/attachment.html>