[Twisted-Python] If the errbacks of a canceled Deferred are called with error other than CancelledError, is this acceptable?

[Terry Jones]

Hi JP

> This is not to say that I believe there is no application that might want
this information, but maybe someone can
> propose some concrete use cases for this information and design can
follow from that.  So far I don't think any
> practical justification to do anything other than `CancelledError` has
been presented.

I pushed to get deferred cancellation in place because I'd wanted it
several times. As a consumer of a deferred-returning API, I wanted to be
able to cancel requests that I had sent off after a certain time. That was
because I was setting up hundreds (or thousands) of network calls and
deferreds that were supposed to run when these completed, etc. If one of
the calls didn't callback or errback for some reason, the whole system
would essentially hang.

So that was one use case, and, as you say, it doesn't need any additional
information.

But, I also built some queuing things where I would have liked extra
information. For example, an app is processing jobs on behalf of a user,
and it might want to be able to cancel jobs in a variety of ways. A user
might indicate that a job should be cancelled. A job might have a certain
timeout and if it doesn't complete in that time then in some cases it
should be retried later. The system might have an adjustable job bandwidth
(like my resizable dispatch queue) and might as a result sometimes need to
cancel jobs that have not started yet (but for which a deferred has been
assigned) or cancel jobs that are actually in progress. These are a
collection of reasons why an app might want to cancel deferreds it has
received or created itself. If there's no way to arrange for any detail on
the reason for cancellation, then things get awkward (you can in some cases
maintain housekeeping info elsewhere, like a dict of deferreds you have
called cancel on that correspond to jobs that need a restart, and check and
clean up that housekeeping info in the errback that handles CancelledError).

In what I originally proposed and implemented in that CancellableDeferred class
(http://twistedmatrix.com/pipermail/twisted-python/2010-January/021298.html)
a deferred could be callbacked or errbacked by anyone who had their hands
on the deferred, with any value of their choosing. So for example my
queuing app might choose to errback a deferred with a CancelledAndResubmit
exception, or similar.

That full flexibility is not what landed in Twisted, though (glyph, and
maybe others, had already done most of an implementation when I arrived at
Pycon that year).

I don't think the ability to callback any deferred you have your hands on
is crucial (though I can think of use cases), but being able to errback is
great, as you can now do, and as you note.

In the mail I sent the other day, I proposed that cancel() could take a
value argument and that that could be given as CancelledError(value) when
the Deferred class triggers the errback, in the case that the cancel method
(if any) for the Deferred instance hasn't already fired the deferred.  That
would give the flexibility I was looking for (as above) for apps to be able
to cancel deferreds and send along any extra information they might
need/want as to why they cancelled it. It would also let apps distinguish
between cases where they cancelled the deferred or something else did.

Terry
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20130620/1b535f5e/attachment.html>