[Twisted-Python] Synchronization techniques
glyph at divmod.com
glyph at divmod.com
Wed Apr 4 14:17:04 EDT 2007
On 05:25 pm, ellisonbg.net at gmail.com wrote:
>The issue brought up in this thread is one that we (the IPython dev
>team) has thought about a lot. There are really two questions about
>having true synchronization primitives (something like d.waitFor() or
>1. Can they be implemented in python/twisted in a reliable and robust
The real question here doesn't involve the words "in python/twisted" :).
>2. Should you use them in your code?
>Unfortunately, the answer to (1) seems to be no - at least not without
>completely refactoring the core of twisted to support this capability
>from the ground level. The tempting way of doing it currently is to
>call reactor.iterate() to spin the event loop at a particular point.
>We have code in IPython that does this:
>[DON'T USE THIS CODE!]
>For very simple things this code works just fine.
I think you mean, "in some cases this code appears to work". Working
"just fine" implies that it is robust and supported. This is an
actively deprecated programming style and there are numerous known
problems (besides the ones you've already documented here) with doing
>d = functionThatReturnsADeferred()
>r = blockOn(d) # spin the reactor until d fires.
>The problem is that if you start to use blockOn in various places in
>your code, a single call to blockOn (which calls reactor.iterate())
>could trigger other calls to blockOn (which will attempt to call
>reactor.iterate() again). The twisted reactor is simply not designed
>to be doubly iterated like this - it leads to all sorts of really
>weird problems that are _impossible_ to track down.
These "weird" problems are the entirely predictable result of violating
every assumption that code makes about its run-time environment when it
is written and tested. An analogous operation would be to write C code
to forcibly delete Python objects rather than go through the garbage
collector because you *really know*, *just this one time*, that you want
to free that memory. If you're careful to never touch that object
again, you might be able to avoid a segfault, but I think most people
would agree that all bets are off at that point.
This might seem like an exaggerated problem, but I have actually seen
code like that more than once written by dyed-in-the-wool C programmers
who didn't "get" how Python's object model worked. It's the same with
dyed-in-the-wool non-concurrent programmers approaching concurrent
systems for the first time.
>This is why people are saying "it can't be done." I should mention
>that it might be possible to implement these things in stackless.
If you were to implement these things in stackless, you would still have
to deal with the inherently problematic issue of apparently "sequential"
code being run recursively when it does not expect to be. In order to
prevent this, you would likely have a completely different programming
model where something properly event-driven, like Twisted itself, were
scheduling "user code" which was written using a different programming
For certain problems such things are a good approach. For example, in
AI code with extremely deeply nested ad-hoc decision trees modeled as if
statements and for loops, the cost of stack ripping becomes high both
conceptually and performance-wise, and it is more natural to model
individual agents as individual control flows (or "cooperative
threads"). This sort of code, though, would be written in a style more
like Erlang, with almost no shared state at all. Part of Twisted's
appeal is that it makes mutable state-sharing between disparate systems
straightforward. In other words, it is a different programming model
for a different set of problems that would require a different pile of
It may well be possible to implement such a layer on top of Twisted, but
there is a curious thing that takes place when people begin to tackle
this problem. Pretty much everyone eventually comes to the realization
that this isn't a good idea for their problem domain, and what they
*actually* want is to wish away the difficulties associated with
concurrency and pretend that they can "just block" and everything will
be OK. The ones who really, really need it (like people dealing with
the aforementioned AI problems) already know their requirements and
quietly go ahead and implement what they need, without any hand-wringing
about how hard programming with Deferreds is or how they'd really like
to block on one.
>So what about (2)? Ignoring the fact that such constructs can't be
>implemented reliably (let's imagine they could be), should you want to
>use them? I think the answer is this:
>The design of Twisted reflects the realities of an asynchronous, event
>driven world where things can and do go wrong when you least expect
>it. The error handling decision tree of Deferreds are a reflection of
>this reality. If you try to make this stuff go away (we have tried ma
>ny times - we are slow learners and very stubborn) you will be
>punished and there will be "weeping and gnashing of teeth." This
>punishment will take the form of buggy code that is difficult to
>maintain and extend.
It sounds like we broadly agree here :).
>With all that said, I have encountered a few highly unusual cases
>where I really did want blockOn to exist. These cases had the
>characteristic that they couldn't be done any other way in Twisted.
>The answer in this case is to ditch twisted and use a tool that is
>better suited to the problem. But in my experience these cases only
>pop up about 0.00001% of the time.
I am very curious about your 0.00001% case. Not that I don't believe
such cases exist, but in every case but one (twisted ticket #2545) the
issue has actually been a documentation problem with Twisted, where it
wasn't clear how to do something the "normal" way with Deferreds and
such. I'd like to know if there is another such doc bug we should be
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Twisted-Python