[Twisted-Python] Some things I've learned: safer callbacks, better t.p.context

[Kevin Conway]

> This is why twisted.python.context came to exist in the first place; I
always wanted to attach it to Deferred somehow

Well, it's not something we've announced yet through any official channel,
but we had to solve the context propagation problem at Atlassian to
instrument our services with traceable logging. We open sourced our
solution at https://bitbucket.org/hipchat/txlocal. The answer for us was an
extension for the reactor, thread pool, and inline callbacks that maintain
the needed state. There's a readme with some insight into how we tool our
services.

We've had it on our backlog to address the mailing list and, possibly even,
discuss what it would take to put this into Twisted. I guess now is as good
if a time as any. Feel free to spin off a another thread or reach out to me
off list with any questions or feedback.

On Tue, Oct 18, 2016, 14:47 Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:

> On Oct 18, 2016, at 5:50 AM, Itamar Turner-Trauring <itamar at itamarst.org>
> wrote:
>
> Not been doing much Twisted lately, but have been doing async stuff
> elsewhere, and I've learned some useful things.
>
>
> Thanks for writing these up, Itamar!  This sort of reflection is rare and
> it's always helpful :).
>
> 1. Callbacks should be sync or async, but never
> sometimes-one-sometimes-the-other. For details go read
> http://blog.ometer.com/2011/07/24/callbacks-synchronous-and-asynchronous/.
> For example, Deferred.addCallback(f) really should never run f()
> immediately.
>
>
> This has come up a lot in a compare-and-contrast of Twisted vs. asyncio.
>
> I agree that the problems with synchronous callbacks are not insignificant
> (reentrancy is a degenerate form of preemption, and as we all know
> preemption is the corrupt wellspring of all bugs).  However, the benefit,
> i.e. consistency of behavior with respect to reentrancy, comes with a cost:
> tight coupling to an event loop.  In asyncio, Future's tight coupling to
> call_soon is a source of problems; it makes it hard to write a test without
> setting up an elaborate scheduling trampoline, whereas
> successResultOf/failureResultOf are quite simple to work with.
>
> I think Deferred as it is today is a pretty good compromise between the
> two positions.  On the one hand it is decoupled from the event loop.  On
> the other - and this is important - *no Deferred-returning API will ever
> call your callbacks synchronously*.  Deferred.addCallback will, of
> course, but savvy Twisted programmers can (and should) do this, if they
> have dependent state changes:
>
> self.manipulateSomeStateForSetup()
>
> d = doSomethingPotentiallySynchronous()
> *self.manipulateSomeStateForProcessing()*
> d.addCallback(completeOperation)
>
>
> As a caller, you can always decide whether you can safely be re-entered or
> not.  In most cases, simply moving the 'addCallback' to the end of the
> function (a-la Go's "defer", oddly enough) is fine.  In more complex cases
> where you really need to unwind reentrancy completely, you can do your own
> callLater(0) or callFromThread() from an object with a reference to a
> reactor.
>
> 3.
>
>
> What happened to '2'? :)
>
> By instrumenting all callbacks it manages, which may or may not
> require item #1, Twisted can have a context that automatically follows
> callbacks. Node has this and it is extremely useful.
>
> http://fredkschott.com/post/2014/02/conquering-asynchronous-context-with-cls/
> is best summary I've found with a bit of searching.
>
>
> This was _always_ supposed to be the way that Twisted worked, but frankly
> I just wasn't smart enough to figure it out.  This is why
> twisted.python.context came to exist in the first place; I always wanted to
> attach it to Deferred somehow.  I will watch this talk intently; if #1
> really is required to address this, my opinion might change.  A PR would be
> intensely appreciated.
>
> -glyph
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20161018/3e000c49/attachment-0002.html>