[Twisted-Python] Twisted in a multicore environment

Werner Thie werner at thieprojects.ch
Sat Jul 14 13:38:16 EDT 2012


On 7/14/12 12:39 AM, gelin yan wrote:
>
>
> On Wed, Jul 11, 2012 at 2:08 AM, Werner Thie <werner at thieprojects.ch
> <mailto:werner at thieprojects.ch>> wrote:
>
>     On 7/10/12 6:17 AM, Laurens Van Houtven wrote:
>      > FWIW,  I have used Ampoule to great effect, but as JP points out
>     it's hardly the only option. You're bound to end up with some
>     measure of multiprocessing. Bear in mind that not all workloads are
>     well-suited for this kind of problem! Always measure before deciding
>     to make your codebase that much more complicated :)
>      >
>      >
>      > cheers
>      > lvh
>      >
>      >
>      >
>      > On 10 Jul 2012, at 18:03, exarkun at twistedmatrix.com
>     <mailto:exarkun at twistedmatrix.com> wrote:
>      >
>      >> On 03:14 pm, augustocaringi at gmail.com
>     <mailto:augustocaringi at gmail.com> wrote:
>      >>> Hi,
>      >>>
>      >>>    I'm researching the best way to implement/use a Twisted-based
>      >>> server in a multicore environment...
>      >>>
>      >>>    There is the Ampoule project, that I realize is considered the
>      >>> best way to do that. Right?
>      >>
>      >> It's a way.  "Best" depends on the details and goals of the project.
>      >>
>      >> Here's a stackoverflow question/answer on basically the same
>     topic.  In
>      >> particular, it specifically answers the question of a listening port
>      >> shared between multiple processes and gives examples of how to
>     do this:
>      >>
>      >> http://bit.ly/MiCHtQ
>      >>
>      >> Jean-Paul
>      >>>    I'm also reading about the internals of Nginx HTTP server. This
>      >>> server utilizes the same reactor pattern of Twisted (epoll
>     based)...
>      >>>
>      >>>    "What resulted is a modular, event-driven, asynchronous,
>      >>> single-threaded, non-blocking architecture which became the
>     foundation
>      >>> of nginx code." http://www.aosabook.org/en/nginx.html
>      >>>
>      >>>    But to maximize the use of processors in a multicore
>     environment,
>      >>> Nginx do this:
>      >>>
>      >>>    "nginx doesn't spawn a process or thread for every connection.
>      >>> Instead, worker processes accept new requests from a shared
>     "listen"
>      >>> socket and execute a highly efficient run-loop inside each
>     worker to
>      >>> process thousands of connections per worker"
>      >>>
>      >>>    My question: There is something similar in Twisted? Or do you
>      >>> think that is easy to implement something like that?
>      >>>
>      >>>    Thanks!
>      >>>
>      >>> --
>      >>> Augusto Mecking Caringi
>     We observed really great scaling on multi cores with moving the
>     application part either to ampoule for PDF production or in the other
>     case I wrote an implementation of self regulating process pool based on
>     spread, leaving only the serving to twisted in both cases.
>
>     With handing work out to other processes you get another benefit which
>     is isolation of python, which is the only way to use a package, like
>     reportlab which survives no sort of reentrancy, for a webservice.
>
>     Werner
>
>     _______________________________________________
>     Twisted-Python mailing list
>     Twisted-Python at twistedmatrix.com
>     <mailto:Twisted-Python at twistedmatrix.com>
>     http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
> Hi Werner
>
>         I want to know whether you have experienced any serious bug with
> ampoule? What version of ampoule did you use? Is it 0.2.0? Thanks.

Hi gelin

Python 2.7.1 Stackless 3.1b3 060516 (release27-maint, Aug  9 2011, 
02:32:21)
[GCC 4.2.1 20070719  [FreeBSD]] on freebsd8
Type "help", "copyright", "credits" or "license" for more information.
 >>> import ampoule
 >>> ampoule.__version__
'0.2.0'
 >>> import twisted
 >>> twisted.__version__
'12.0.0'

sitting in production now for almost six month, a webservice for 
template based PDF production with reportlab & z3c.rml.

The only maintenance I do is creating more templates, never done a 
restart so far, logs are clean.

As painless as one could wish for

Werner



More information about the Twisted-Python mailing list