[Twisted-Python] Using reactor with multiple processes

David Yoakley david.yoakley at gmail.com
Fri Sep 18 13:01:11 EDT 2009

I promised a follow-up on what we found so here it is.  As Phil
indicated prior and is now clear to us, twisted is not designed to be
fork safe.  But as I considered patching it to set up all state when
the reactor is started then I began to worry about whether I was
really going to be happy with using twisted in a python multiprocess
architecture since its ultimately not twisted aware and thus there are
probably going to be blocking issues with using any of cool
multiprocess abstractions.  I think we were just going in the wrong
direction, attracted by a shiny new thing.  To make a long story
short, we think making twisted work with python multiprocess package
is probably a dead end.  The abstractions are nice and maybe the right
direction but under the hood its just not going to work.

Instead, we implemented a simple framework using twisted
PerspectiveBroker (pb) where we spawn processes with python
subprocess.Popen and use PB remote calls to manage the processes.  Its
now working well and is very simple.

There are other aspects we didn't think about that are actually
working out better for us with this PB based framework.  Mostly, our
child processes are much less coupled to the parent; they are able to
be launched on their own from a command line for unit tests and such
but each child is an independent server (implementing
pb.PBServerFactory).  They can be launched from our main process
(implementing pb.PBClientFactory) in production and any number of
other ways during development and test.  Similarly, we could allocate
the children even to separate machines if we wanted to since their
connections are via tcp.  The last thing is more subtle.  We are now
not tempted to use tighter synchronization (more coupling) such as
shared memory or other mechanisms offered by python multiprocess in
cases where its not strictly necessary.   The processes all share
configuration (such as child tcp port assignments) by having the
parent send on the command line,  an absolute path to a common config
file when the process is launched and after that the children carry on
pretty much independently, occasionally collaborating with each other
and with the parent through remote calls as needed.  When its time for
the application to go away, the parent sends each child a stop command
which blocks the parents shutdown completion (in the sense that its
synchronous and the parent waits on the response, but NOT in the sense
that any other logical twisted threads on the parent get blocked)
until the child responds or times out.

David Yoakley

More information about the Twisted-Python mailing list