[Twisted-Python] Twisted in a multicore environment

[Werner Thie]

On 7/10/12 6:17 AM, Laurens Van Houtven wrote:
> FWIW,  I have used Ampoule to great effect, but as JP points out it's hardly the only option. You're bound to end up with some measure of multiprocessing. Bear in mind that not all workloads are well-suited for this kind of problem! Always measure before deciding to make your codebase that much more complicated :)
>
>
> cheers
> lvh
>
>
>
> On 10 Jul 2012, at 18:03, exarkun at twistedmatrix.com wrote:
>
>> On 03:14 pm, augustocaringi at gmail.com wrote:
>>> Hi,
>>>
>>>    I'm researching the best way to implement/use a Twisted-based
>>> server in a multicore environment...
>>>
>>>    There is the Ampoule project, that I realize is considered the
>>> best way to do that. Right?
>>
>> It's a way.  "Best" depends on the details and goals of the project.
>>
>> Here's a stackoverflow question/answer on basically the same topic.  In
>> particular, it specifically answers the question of a listening port
>> shared between multiple processes and gives examples of how to do this:
>>
>>   http://bit.ly/MiCHtQ
>>
>> Jean-Paul
>>>    I'm also reading about the internals of Nginx HTTP server. This
>>> server utilizes the same reactor pattern of Twisted (epoll based)...
>>>
>>>    "What resulted is a modular, event-driven, asynchronous,
>>> single-threaded, non-blocking architecture which became the foundation
>>> of nginx code." http://www.aosabook.org/en/nginx.html
>>>
>>>    But to maximize the use of processors in a multicore environment,
>>> Nginx do this:
>>>
>>>    "nginx doesn't spawn a process or thread for every connection.
>>> Instead, worker processes accept new requests from a shared "listen"
>>> socket and execute a highly efficient run-loop inside each worker to
>>> process thousands of connections per worker"
>>>
>>>    My question: There is something similar in Twisted? Or do you
>>> think that is easy to implement something like that?
>>>
>>>    Thanks!
>>>
>>> --
>>> Augusto Mecking Caringi
We observed really great scaling on multi cores with moving the 
application part either to ampoule for PDF production or in the other 
case I wrote an implementation of self regulating process pool based on 
spread, leaving only the serving to twisted in both cases.

With handing work out to other processes you get another benefit which 
is isolation of python, which is the only way to use a package, like 
reportlab which survives no sort of reentrancy, for a webservice.

Werner