[Twisted-Python] Converting async to sync code (blocking until deferreds fire)

[Phil Mayers]

Jack Whitham wrote:
> Hi,
> 
> What is the best way to convert asynchronous code (non-blocking,
> returning deferreds) to synchronous code (blocking)? 

Don't.

> 
> I have a client that is written using Twisted. Most of the methods
> within the client object return Deferreds because they send messages
> to a server and replies may take some time. This is ideal for the 
> wxwidgets GUI that was originally intended to use the client. But
> now I want to write test scripts for the server that are synchronous.
> 
> For example, the client provides the method "GetBitInfo" which returns
> a deferred. The deferred is called back with a list as its parameter
> after the server responds to a request. I would like to be able to 
> do something like this:
> 
>> def GotBitInfo(l): pass
>>
>> if __name__ == "__main__":
>>     c = Client()
>>
>>     d = c.GetBitInfo()
>>     d.addCallback(GotBitInfo)
>>     waitForDeferred(d)
>>
>>     d = c.DoSomethingElse()
> 
> In other words, how do I waitForDeferred(), i.e. block until a Deferred fires?

You've asked a complicated question, to which I'm sure you'll get other 
answers. A very short answer is "you don't, the reactor must not block". 
There are ways to achieve this, most of which make it *look* like the 
reactor blocked when it really didn't.

I had the same desire at one point, to make the code "look" simple so 
that others could use it. I have come to the conclusion that was a 
mistake; trying to hide the underlying async behaviour can actually be 
counter-productive.

You could use inlineCallbacks and python2.5 generators, which is a style 
I swing between really liking and worrying about:

from twisted.internet import defer

@defer.inlineCallbacks
def main():
     c = Client()

     # execution of the generator is paused here until the
     # deferred you've just yielded is callback'ed
     r = yield c.GetBitInfo()

     if r:
         # "r" now contains the value of the callback
         pass

     # optionally
     reactor.callLater(0.1, reactor.stop)

if __name__=='__main__':
     from twisted.internet import reactor
     reactor.addSystemEventTrigger('after', 'startup', main)
     reactor.run()

There are other approaches using Stackless, greenlets and running the 
reactor in a thread (shudder). For greenlets you might look at:

https://launchpad.net/corotwine/