[Twisted-Python] multiprocessing capability?
Darren Govoni
darren at ontrenet.com
Thu Feb 25 20:33:52 EST 2010
What you refer to is different than what I need. The real 'Process'
implementation is new to Python 2.6
http://docs.python.org/library/multiprocessing.html
and is not supported in Twisted at the moment. The Process or threads in
Twisted now, use Python threading/process constructs outside of the new
multiprocessing module, will suffer from the Python GIL limitations -
which hinders higher performance computing.
It works, sure. But its not what I'm asking about.
However, I found a workaround for now.
Ideally, what I wanted to do was use something like
threads.deferToPool() or similar, that _would_ use Python's new support
for OS processes.
Someone wrote a wrapper to this on the net, but I was curious if this
will be supported in Twisted.
Darren
On Fri, 2010-02-26 at 01:13 +0000, Johann Borck wrote:
> Darren Govoni wrote:
> > The nice thing about using Python's process support is that you can
> > spawn native Processes that run in separate heaps directly from ONE
> > Python Twisted app. Not many running side-by-side, which adds the
> > complexity of now coordinating among them (however easy with
> > additional protocols like spread).
> >
> > Inter-Process communication is also supported in Python's new
> > multiprocessing package. And again, it can all be orchestrated from a
> > single service _instance_.
> >
> > In my code, I need to run "on the metal" for some tasks and not
> > others. Agreed, the event-based reactor threading in Twisted is great.
> > But not for all modes of computation. For those, I offload onto OS
> > processes directly onto CPU cores. Twisted does not provide a way to
> > leverage its API against Python's support for this feature.
> But twisted provides this feature *itself*, and has done so long before
> the multiprocessing module existed - look at Process Protocol
> http://twistedmatrix.com/documents/current/core/howto/process.html and
> the stdio stuff:
> http://twistedmatrix.com/documents/current/core/examples/stdiodemo.py,
> or, as has been mentioned before, ampoule. Of course, if you get
> multiprocess to work with twisted, that's fine, but you're probably
> unnecessarily adding complexity to your application while substracting
> compatibility with python versions before 2.6 for no good reason, at
> least none you mentioned so far.
>
> regards,
> Johann
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20100225/3e79c305/attachment.htm
More information about the Twisted-Python
mailing list