[Twisted-Python] Synchronization techniques

[Brian Granger]

The issue brought up in this thread is one that we (the IPython dev
team) has thought about a lot.  There are really two questions about
having true synchronization primitives (something like d.waitFor() or
blockOn(d)):

1.  Can they be implemented in python/twisted in a reliable and robust way?

2.  Should you use them in your code?

Unfortunately, the answer to (1) seems to be no - at least not without
completely refactoring the core of twisted to support this capability
from the ground level.  The tempting way of doing it currently is to
call reactor.iterate() to spin the event loop at a particular point.
We have code in IPython that does this:

http://projects.scipy.org/ipython/ipython/browser/ipython/branches/saw/ipython1/kernel/blockon.py
[DON'T USE THIS CODE!]

For very simple things this code works just fine.

d = functionThatReturnsADeferred()
r = blockOn(d)          # spin the reactor until d fires.

The problem is that if you start to use blockOn in various places in
your code, a single call to blockOn (which calls reactor.iterate())
could trigger other calls to blockOn (which will attempt to call
reactor.iterate() again).  The twisted reactor is simply not designed
to be doubly iterated like this - it leads to all sorts of really
weird problems that are _impossible_ to track down.

This is why people are saying "it can't be done."  I should mention
that it might be possible to implement these things in stackless.

So what about (2)?  Ignoring the fact that such constructs can't be
implemented reliably (let's imagine they could be), should you want to
use them?  I think the answer is this:

The design of Twisted reflects the realities of an asynchronous, event
driven world where things can and do go wrong when you least expect
it.  The error handling decision tree of Deferreds are a reflection of
this reality.  If you try to make this stuff go away (we have tried ma
ny times - we are slow learners and very stubborn) you will be
punished and there will be "weeping and gnashing of teeth."  This
punishment will take the form of buggy code that is difficult to
maintain and extend.

With all that said, I have encountered a few highly unusual cases
where I really did want blockOn to exist.  These cases had the
characteristic that they couldn't be done any other way in Twisted.
The answer in this case is to ditch twisted and use a tool that is
better suited to the problem.  But in my experience these cases only
pop up about 0.00001% of the time.

Brian