[Twisted-Python] Synchronization techniques

Paul G paul-lists at perforge.com
Wed Apr 4 13:34:32 EDT 2007

Daniel Miller wrote:
-- snip --
> So anyway, I rewrote my server-side library to do it the twisted way
> and return deferred's instead trying rig up some way of waiting for
> them. I still think it would be super useful to be able to pseudo-
> block on a deferred (i.e. allow the reactor to process other events
> while waiting for the deferred). It is very annoying to have to
> rewrite many layers of code when twisted is introduced into a
> program. I did find gthreadless.py, and maybe that would do it.
> Unfortunately discussion on that seems to have been dropped some time
> ago...

you can still use that method *if* you're willing to patch your cpython with 
the stackless changes and live with the requirement that this be done on all 
cpythons your code is going to run on. short of that, or using evil bytecode 
hacks, it's not really possible to implement - you need the ability to save, 
switch and restore stacks plus call into the middle of a function. while 
this can be done with generators, it is virtually equivalent to splitting 
your functions into a top/bottom half and is not directly supported, so it 
ends up looking ugly. if pypy ever gets to the point where it's usable in 
production (has the required c extensions ported and runs close in speed to 
cpython on non-trivial code), syntactic support for such usage could be 
added. please note that if you hadn't wanted to take advantage of being in 
an event loop, you could have easily stuck your code - unchanged - into a 
thread pool (via apis helpfully provided by twisted) and just written a few 
lines of code to adapt the interfaces.

this has little to do with twisted, which - imo - does a wonderful job of 
making the (for some, abeit not me personally) difficult execution flow 
model palatable and clear. twisted might be a very good framework, but it 
isn't an async fairy that sneaks into your room at night and - if you've 
been good - rewrites your code to be all twisty and eventful and leaves a 
quarter under your pillow.

with all that said, the sooner you start thinking of a program as a series 
of events being emitted and handled, the better for you and your twisted 
code - everything will become very natural at that point instead of looking 
like an idiosyncracy. while there are a few languages, like erlang for 
example, that express event loops with cooperative threading syntax 
(processes, messages and futures), i'm of the opinion that it's actually 
harmful to hide the event loop details from people who haven't started 
thinking that way yet - they should be simply syntactic sugar for when 
you've already grokked the model. it's inherently impossible to write good 
concurrent code without this insight, including preemptiver multithreading 
code as well. writing preemption and smp safe mt code boils down to 
understanding the chain of processing each 'event', where such event is 
access to shared data, goes through and the interaction between two or more 
such events descending down this chain - this is the only effective way i've 
found to think about memory barriers and cache coherency for example. just 
like grokking functional programming, grokking this will make you (the 
impersonal pronoun version) a better programmer (fwiw, it's made functional 
programming easier to get for me personally).


More information about the Twisted-Python mailing list