[Twisted-Python] multiprocessing capability?

Darren Govoni darren at ontrenet.com
Thu Feb 25 18:33:52 MST 2010


What you refer to is different than what I need. The real 'Process'
implementation is new to Python 2.6
http://docs.python.org/library/multiprocessing.html
and is not supported in Twisted at the moment. The Process or threads in
Twisted now, use Python threading/process constructs outside of the new
multiprocessing module, will suffer from the Python GIL limitations -
which hinders higher performance computing.
It works, sure. But its not what I'm asking about.

However, I found a workaround for now.

Ideally, what I wanted to do was use something like
threads.deferToPool() or similar, that _would_ use Python's new support
for OS processes.
Someone wrote a wrapper to this on the net, but I was curious if this
will be supported in Twisted.

Darren

On Fri, 2010-02-26 at 01:13 +0000, Johann Borck wrote:

> Darren Govoni wrote:
> > The nice thing about using Python's process support is that you can 
> > spawn native Processes that run in separate heaps directly from ONE 
> > Python Twisted app. Not many running side-by-side, which adds the 
> > complexity of now coordinating among them (however easy with 
> > additional protocols like spread).
> >
> > Inter-Process communication is also supported in Python's new 
> > multiprocessing package. And again, it can all be orchestrated from a 
> > single service _instance_.
> >
> > In my code, I need to run "on the metal" for some tasks and not 
> > others. Agreed, the event-based reactor threading in Twisted is great.
> > But not for all modes of computation. For those, I offload onto OS 
> > processes directly onto CPU cores. Twisted does not provide a way to 
> > leverage its API against Python's support for this feature. 
> But twisted provides this feature *itself*, and has done so long before 
> the multiprocessing module existed - look at Process Protocol 
> http://twistedmatrix.com/documents/current/core/howto/process.html and 
> the stdio stuff: 
> http://twistedmatrix.com/documents/current/core/examples/stdiodemo.py, 
> or, as has been mentioned before, ampoule. Of course, if you get 
> multiprocess to work with twisted, that's fine, but you're probably 
> unnecessarily adding complexity to your application while substracting 
> compatibility with python versions  before 2.6 for no good reason, at 
> least none you mentioned so far.
> 
> regards,
> Johann
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20100225/3e79c305/attachment.html>


More information about the Twisted-Python mailing list