[Twisted-Python] Twisted in a multicore environment

gelin yan dynamicgl at gmail.com
Sun Jul 15 10:52:05 EDT 2012


On Sun, Jul 15, 2012 at 1:38 AM, Werner Thie <werner at thieprojects.ch> wrote:

> On 7/14/12 12:39 AM, gelin yan wrote:
> >
> >
> > On Wed, Jul 11, 2012 at 2:08 AM, Werner Thie <werner at thieprojects.ch
> > <mailto:werner at thieprojects.ch>> wrote:
> >
> >     On 7/10/12 6:17 AM, Laurens Van Houtven wrote:
> >      > FWIW,  I have used Ampoule to great effect, but as JP points out
> >     it's hardly the only option. You're bound to end up with some
> >     measure of multiprocessing. Bear in mind that not all workloads are
> >     well-suited for this kind of problem! Always measure before deciding
> >     to make your codebase that much more complicated :)
> >      >
> >      >
> >      > cheers
> >      > lvh
> >      >
> >      >
> >      >
> >      > On 10 Jul 2012, at 18:03, exarkun at twistedmatrix.com
> >     <mailto:exarkun at twistedmatrix.com> wrote:
> >      >
> >      >> On 03:14 pm, augustocaringi at gmail.com
> >     <mailto:augustocaringi at gmail.com> wrote:
> >      >>> Hi,
> >      >>>
> >      >>>    I'm researching the best way to implement/use a Twisted-based
> >      >>> server in a multicore environment...
> >      >>>
> >      >>>    There is the Ampoule project, that I realize is considered
> the
> >      >>> best way to do that. Right?
> >      >>
> >      >> It's a way.  "Best" depends on the details and goals of the
> project.
> >      >>
> >      >> Here's a stackoverflow question/answer on basically the same
> >     topic.  In
> >      >> particular, it specifically answers the question of a listening
> port
> >      >> shared between multiple processes and gives examples of how to
> >     do this:
> >      >>
> >      >> http://bit.ly/MiCHtQ
> >      >>
> >      >> Jean-Paul
> >      >>>    I'm also reading about the internals of Nginx HTTP server.
> This
> >      >>> server utilizes the same reactor pattern of Twisted (epoll
> >     based)...
> >      >>>
> >      >>>    "What resulted is a modular, event-driven, asynchronous,
> >      >>> single-threaded, non-blocking architecture which became the
> >     foundation
> >      >>> of nginx code." http://www.aosabook.org/en/nginx.html
> >      >>>
> >      >>>    But to maximize the use of processors in a multicore
> >     environment,
> >      >>> Nginx do this:
> >      >>>
> >      >>>    "nginx doesn't spawn a process or thread for every
> connection.
> >      >>> Instead, worker processes accept new requests from a shared
> >     "listen"
> >      >>> socket and execute a highly efficient run-loop inside each
> >     worker to
> >      >>> process thousands of connections per worker"
> >      >>>
> >      >>>    My question: There is something similar in Twisted? Or do you
> >      >>> think that is easy to implement something like that?
> >      >>>
> >      >>>    Thanks!
> >      >>>
> >      >>> --
> >      >>> Augusto Mecking Caringi
> >     We observed really great scaling on multi cores with moving the
> >     application part either to ampoule for PDF production or in the other
> >     case I wrote an implementation of self regulating process pool based
> on
> >     spread, leaving only the serving to twisted in both cases.
> >
> >     With handing work out to other processes you get another benefit
> which
> >     is isolation of python, which is the only way to use a package, like
> >     reportlab which survives no sort of reentrancy, for a webservice.
> >
> >     Werner
> >
> >     _______________________________________________
> >     Twisted-Python mailing list
> >     Twisted-Python at twistedmatrix.com
> >     <mailto:Twisted-Python at twistedmatrix.com>
> >     http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> >
> >
> > Hi Werner
> >
> >         I want to know whether you have experienced any serious bug with
> > ampoule? What version of ampoule did you use? Is it 0.2.0? Thanks.
>
> Hi gelin
>
> Python 2.7.1 Stackless 3.1b3 060516 (release27-maint, Aug  9 2011,
> 02:32:21)
> [GCC 4.2.1 20070719  [FreeBSD]] on freebsd8
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> import ampoule
>  >>> ampoule.__version__
> '0.2.0'
>  >>> import twisted
>  >>> twisted.__version__
> '12.0.0'
>
> sitting in production now for almost six month, a webservice for
> template based PDF production with reportlab & z3c.rml.
>
> The only maintenance I do is creating more templates, never done a
> restart so far, logs are clean.
>
> As painless as one could wish for
>
> Werner
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>

Hi Werner

     Thanks for your valuable info. It looks like you are using stackless
python with twisted. Have you taken any benefit from both of them?
Stackless is awesome but I have no idea what is the better way to work with
twisted. Thanks.

Regards

gelin yan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20120715/02fdfb35/attachment-0001.htm 


More information about the Twisted-Python mailing list