[Twisted-Python] multiprocessing capability?

Darren Govoni darren at ontrenet.com
Thu Feb 25 16:43:52 EST 2010


The nice thing about using Python's process support is that you can
spawn native Processes that run in separate heaps directly from ONE
Python Twisted app. Not many running side-by-side, which adds the
complexity of now coordinating among them (however easy with additional
protocols like spread).

Inter-Process communication is also supported in Python's new
multiprocessing package. And again, it can all be orchestrated from a
single service _instance_.

In my code, I need to run "on the metal" for some tasks and not others.
Agreed, the event-based reactor threading in Twisted is great.
But not for all modes of computation. For those, I offload onto OS
processes directly onto CPU cores. Twisted does not provide a way to
leverage its API against Python's support for this feature. So I have to
find a way to marry the two.

What I ended up doing was using the multiprocess package to kick off
hard Process objects in a Python Process pool executing Python
functions.

Those functions make calls into Twisted, but for it to work, they had to
start their own reactors because a Process has its own, separate OS
memory, etc. Running compute intensive tasks in processes with their own
memory makes a lot of sense for some things that Python cannot do with
virtual machine thread contexts.

Darren

On Thu, 2010-02-25 at 10:21 -0800, K. Richard Pixley wrote:

> Single threaded, event loop based code like twisted rocks hard.
> 
> Once upon a time, threads were like that too and the distinction
> between threads and event loops was grey.  But with the advent of
> mandatory preemptive thread scheduling and the ability to run multiple
> threads on separate shared memory processors, the difference between
> programming with threads and programming with parallel heavy weight
> processes that share memory became extremely grey, (aside from the
> problems debugging threads which don't exist for heavy weight
> processes).
> 
> Threads routinely use shared memory and shared memory (generally)
> requires a common kernel.  OTOH, message passing can use a common
> kernel but can also extend out to other machines on the network.  If
> you use twisted for highly efficient "single thread/multiple task"
> heavy weight processes, and something like spread, you end up with the
> best of all worlds.  Highly efficient, symmetric, network based
> parallelism, with fault tolerance thrown in for free.
> 
> My point here is that there are other ways to go about exploiting
> symmetric multiprocessor machines, even banks of them, that neither
> require threads, nor the multiprocessing package.
> 
> --rich
> 
> Darren Govoni wrote: 
> 
> > Looks interesting. I'm going to check out that package.
> > 
> > 
> > My original request was more along the lines of using Python's new
> > support for native CPU core's and processes (the multiprocessing
> > package is for this). Python's built-in thread support has global
> > lock constraints that underperform in some situations.
> > 
> > But I ran into a problem using multiprocessing module with Twisted
> > that was pointed out on the Twisted trac with pickling class methods
> > and apparently Python's CPU threading support attempts to do this in
> > some situations (e.g. when I try to pass a class method to a native
> > thread).
> > 
> > On Wed, 2010-02-24 at 12:04 -0800, K. Richard Pixley wrote:
> > 
> > > Glyph Lefkowitz wrote: 
> > > 
> > > > On Feb 21, 2010, at 8:00 PM, Alexandre Quessy wrote
> > > >       
> > > > 
> > > > > Hello everyone,
> > > > > I have done something similar to this, but I used the children IO
> > > > > stream to control them. Maybe I should have done that using some
> > > > > higher level protocol, such as AMP or PB.
> > > > >     
> > > > >         
> > > > 
> > > > Using a higher-level protocol is generally better, if for no other reason that it gives you a framework within which to document your design decisions.  It's much easier to say "An AMP command with a 'foo' String argument and a 'bar' Integer argument" than to say "The first two bytes of the message are the length of the first argument.  The next n bytes are the first argument.  The first argument shall be interpreted as... (etc, etc)"
> > > >       
> > > 
> > > I'm working on an interface right now to the spread toolkit,
> > > (http://spread.org), which implements virtual synchrony,
> > > (http://en.wikipedia.org/wiki/Virtual_synchrony).
> > > 
> > > For distributed, symmetric, fault tolerant parallelism in small to
> > > medium scale with high reliability, this might be an option.
> > > 
> > > --rich 
> > > 
> > > _______________________________________________
> > > Twisted-Python mailing list
> > > Twisted-Python at twistedmatrix.com
> > > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> > >     
> > 
> > 
> > 
> > 
> > ____________________________________________________________________
> > 
> > _______________________________________________
> > Twisted-Python mailing list
> > Twisted-Python at twistedmatrix.com
> > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> >   
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20100225/0dc7336e/attachment-0001.htm 


More information about the Twisted-Python mailing list