[Twisted-Python] A pseudo-deferred class that can be canceled

Glyph Lefkowitz glyph at twistedmatrix.com
Wed Jan 6 22:32:08 EST 2010

On Jan 6, 2010, at 7:09 AM, Terry Jones wrote:

>>>>>> "Glyph" == Glyph Lefkowitz <glyph at twistedmatrix.com> writes:
> Glyph> On Jan 5, 2010, at 4:53 PM, Terry Jones wrote:
> Glyph> I understand what you're saying: you're interested in a subset of
> Glyph> what I'm interested in, here.
> Yes.

> Glyph> I think that coming up with a good API and semantics for "I no
> Glyph> longer care about the result here" has a huge amount of overlap with
> Glyph> this anyway.
> Hmmmm. I'm not sure about that. I agree if you s/huge amount/tiny/ :-)

Okay, "a huge amount" was not usefully descriptive :).

What I mean is, there are a lot of weird little edge-cases in how multiple layers of the stack interact when they're dealing with a shared Deferred, and if we're 

However, upon further inspection I think that they key distinction between what you've proposed and what I'm talking about is the distinction between cancelling *one* layer of the callback chain and cancelling *all* layers of the callback chain.

Your description (elided for brevity's sake) was very helpful.  You've got resources which your callbacks are consuming by way of being "currently outstanding", and you want to be able to free *those* resources, without necessarily worrying about 

> Yes, agreed. I like the fact that the class is simple and that it deals
> with the client-side issues, allowing ignoring, timing out, early firing,
> etc.  As you say, the much harder problem remains. But the harder problem
> is a bit less messy now (at least in my mind): it's "just" cancellation.
> Responsibilities are cleanly divided by my class - the client takes care of
> itself, and cancellation has *only* to deal with callbacks placed on a
> deferred that was generated by what the client called.

I don't think that you can completely separate the problems.  You seem to have a reasonable solution to the problem of one layer of the Deferred stack, but once you're trying to deal with multiple layers of the stack at once, interactions occur which can be difficult to reconcile with the same API, many of which are already documented in the ticket's discussion.

> Looked at from this POV, an approach to cancellation would be for code that
> is able to cancel operations it has begun to also provide a cancel method.
> One way to think about doing this would be to have the cancel method take a
> deferred as an argument.

This is a *very* interesting idea, although I don't like the API that you propose for it.  By separating the cancel method from the Deferred itself, you remove the ability for a trivial client of that Deferred to say "forget about it" without also maintaining a reference to the thing that gave it the Deferred in the first place.  That means that you need a new 'operation' API, and your code needs to take twice as many parameters, and it generally gets ugly.

> Something like my class could then hand the
> deferred back, effectively saying "my client is no longer interested in
> this deferred. You can call/errback it, or not, it makes no difference to
> us". If you've done that once, you can do it multiple times - by which I
> mean that I might write code that's a client of getPage, and getPage is a
> client of XXX, and XXX is a client of YYY, etc. Each could in turn pass the
> deferred it got back to the thing that created it.

This implies, to me, that the cancellation callback would be better passed to addCallbacks(): effectively creating a third callback chain going from invoker to responder rather than the other way 'round as callbacks and errbacks do.

I have stumbled in the direction of this thought a few times already but this is the first time I've had a really clear grasp of how it would work.  Now I can see that each layer of the stack may have its own resources that it might want to clean up... previously I thought this could be done entirely with errbacks, but in this version, it doesn't matter if the base deferred doesn't know how to kick off the errback chain: all the resources on the *rest* of the callback chain can be cleaned up.

I'm going to need to figure out some good values for XXX and YYY here in order to truly dispel the fog, though.  The examples you provided are good but I still don't have a good feel for what might be a good general description of what resources could be used in different parts of the callback chain.

> If there's no cancel method, then that's as far as can be gone with canceling.

This is one of the really tricky issues that has faced this feature all along: what happens when some part of the chain involved doesn't know what to do with a canceller?  And your solution here seems like it may be a very elegant hack: do exactly the same thing as other parts of the callback chain.  What I mean is: currently, if a particular callback pair doesn't have a callback or an errback, the behavior is to do nothing and pass the result through.  Cancellation could do exactly the same thing!

> At that point the
> result is no longer passed because the first ControllableDeferred instance
> that's involved will effectively snip the link (or send an early result) in
> the sequence of steps that would originally have been done.

Severing the link seems like a problem though; if we do that, then introducing any non-cancellation-aware Deferred - or callback, for that matter - into a cancellation-aware pipeline will prevent cancellations from propagating further up, and there should be no reason to do that.

> And it keeps all code for doing cancellation out of the Deferred class.

Why is it that you want to keep the cancellation code out of Deferred?  It seems very useful to me to have one object that you can say "stop" to, without necessarily knowing what's going on above it or where it came from.

> OK, sorry for so many words. I hope this seems like it's heading in a
> useful direction. It does to me.

Yes, this has been very useful.  I hope we can distill this into some useful conclusions soon. :)

More information about the Twisted-Python mailing list