[Twisted-Python] multiprocessing capability?

[Alexandre Quessy]

Hello !

2010/2/25 Johann Borck <johann.borck at densedata.com>:
> Darren Govoni wrote:
>> The nice thing about using Python's process support is that you can
>> spawn native Processes that run in separate heaps directly from ONE
>> Python Twisted app. Not many running side-by-side, which adds the
>> complexity of now coordinating among them (however easy with
>> additional protocols like spread).
>>
>> Inter-Process communication is also supported in Python's new
>> multiprocessing package. And again, it can all be orchestrated from a
>> single service _instance_.
>>
>> In my code, I need to run "on the metal" for some tasks and not
>> others. Agreed, the event-based reactor threading in Twisted is great.
>> But not for all modes of computation. For those, I offload onto OS
>> processes directly onto CPU cores. Twisted does not provide a way to
>> leverage its API against Python's support for this feature.
> But twisted provides this feature *itself*, and has done so long before
> the multiprocessing module existed - look at Process Protocol
> http://twistedmatrix.com/documents/current/core/howto/process.html and
> the stdio stuff:
> http://twistedmatrix.com/documents/current/core/examples/stdiodemo.py,

That's exactly what I have done with my application Lunch, and I am
very satisfied with the results. It can be distributed on many hosts
via SSH ! That comes free with standard IO and process management. You
can get the source code from http://svn.sat.qc.ca/trac/lunch

Cheers,
Alexandre


> or, as has been mentioned before, ampoule. Of course, if you get
> multiprocess to work with twisted, that's fine, but you're probably
> unnecessarily adding complexity to your application while substracting
> compatibility with python versions  before 2.6 for no good reason, at
> least none you mentioned so far.
>
> regards,
> Johann
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>



-- 
Alexandre Quessy
http://alexandre.quessy.net/