[Twisted-Python] Some things I've learned: safer callbacks, better t.p.context

Glyph Lefkowitz glyph at twistedmatrix.com
Tue Oct 18 13:44:33 MDT 2016


> On Oct 18, 2016, at 5:50 AM, Itamar Turner-Trauring <itamar at itamarst.org> wrote:
> 
> Not been doing much Twisted lately, but have been doing async stuff
> elsewhere, and I've learned some useful things.

Thanks for writing these up, Itamar!  This sort of reflection is rare and it's always helpful :).

> 1. Callbacks should be sync or async, but never
> sometimes-one-sometimes-the-other. For details go read
> http://blog.ometer.com/2011/07/24/callbacks-synchronous-and-asynchronous/.
> For example, Deferred.addCallback(f) really should never run f()
> immediately.

This has come up a lot in a compare-and-contrast of Twisted vs. asyncio.

I agree that the problems with synchronous callbacks are not insignificant (reentrancy is a degenerate form of preemption, and as we all know preemption is the corrupt wellspring of all bugs).  However, the benefit, i.e. consistency of behavior with respect to reentrancy, comes with a cost: tight coupling to an event loop.  In asyncio, Future's tight coupling to call_soon is a source of problems; it makes it hard to write a test without setting up an elaborate scheduling trampoline, whereas successResultOf/failureResultOf are quite simple to work with.

I think Deferred as it is today is a pretty good compromise between the two positions.  On the one hand it is decoupled from the event loop.  On the other - and this is important - no Deferred-returning API will ever call your callbacks synchronously.  Deferred.addCallback will, of course, but savvy Twisted programmers can (and should) do this, if they have dependent state changes:

self.manipulateSomeStateForSetup()
d = doSomethingPotentiallySynchronous()
self.manipulateSomeStateForProcessing()
d.addCallback(completeOperation)

As a caller, you can always decide whether you can safely be re-entered or not.  In most cases, simply moving the 'addCallback' to the end of the function (a-la Go's "defer", oddly enough) is fine.  In more complex cases where you really need to unwind reentrancy completely, you can do your own callLater(0) or callFromThread() from an object with a reference to a reactor.

> 3.

What happened to '2'? :)

> By instrumenting all callbacks it manages, which may or may not
> require item #1, Twisted can have a context that automatically follows
> callbacks. Node has this and it is extremely useful.
> http://fredkschott.com/post/2014/02/conquering-asynchronous-context-with-cls/
> is best summary I've found with a bit of searching.

This was _always_ supposed to be the way that Twisted worked, but frankly I just wasn't smart enough to figure it out.  This is why twisted.python.context came to exist in the first place; I always wanted to attach it to Deferred somehow.  I will watch this talk intently; if #1 really is required to address this, my opinion might change.  A PR would be intensely appreciated.

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20161018/d234ff8f/attachment-0002.html>


More information about the Twisted-Python mailing list