[Twisted-Python] Twisted in a multicore environment
werner at thieprojects.ch
Sat Jul 14 13:38:16 EDT 2012
On 7/14/12 12:39 AM, gelin yan wrote:
> On Wed, Jul 11, 2012 at 2:08 AM, Werner Thie <werner at thieprojects.ch
> <mailto:werner at thieprojects.ch>> wrote:
> On 7/10/12 6:17 AM, Laurens Van Houtven wrote:
> > FWIW, I have used Ampoule to great effect, but as JP points out
> it's hardly the only option. You're bound to end up with some
> measure of multiprocessing. Bear in mind that not all workloads are
> well-suited for this kind of problem! Always measure before deciding
> to make your codebase that much more complicated :)
> > cheers
> > lvh
> > On 10 Jul 2012, at 18:03, exarkun at twistedmatrix.com
> <mailto:exarkun at twistedmatrix.com> wrote:
> >> On 03:14 pm, augustocaringi at gmail.com
> <mailto:augustocaringi at gmail.com> wrote:
> >>> Hi,
> >>> I'm researching the best way to implement/use a Twisted-based
> >>> server in a multicore environment...
> >>> There is the Ampoule project, that I realize is considered the
> >>> best way to do that. Right?
> >> It's a way. "Best" depends on the details and goals of the project.
> >> Here's a stackoverflow question/answer on basically the same
> topic. In
> >> particular, it specifically answers the question of a listening port
> >> shared between multiple processes and gives examples of how to
> do this:
> >> http://bit.ly/MiCHtQ
> >> Jean-Paul
> >>> I'm also reading about the internals of Nginx HTTP server. This
> >>> server utilizes the same reactor pattern of Twisted (epoll
> >>> "What resulted is a modular, event-driven, asynchronous,
> >>> single-threaded, non-blocking architecture which became the
> >>> of nginx code." http://www.aosabook.org/en/nginx.html
> >>> But to maximize the use of processors in a multicore
> >>> Nginx do this:
> >>> "nginx doesn't spawn a process or thread for every connection.
> >>> Instead, worker processes accept new requests from a shared
> >>> socket and execute a highly efficient run-loop inside each
> worker to
> >>> process thousands of connections per worker"
> >>> My question: There is something similar in Twisted? Or do you
> >>> think that is easy to implement something like that?
> >>> Thanks!
> >>> --
> >>> Augusto Mecking Caringi
> We observed really great scaling on multi cores with moving the
> application part either to ampoule for PDF production or in the other
> case I wrote an implementation of self regulating process pool based on
> spread, leaving only the serving to twisted in both cases.
> With handing work out to other processes you get another benefit which
> is isolation of python, which is the only way to use a package, like
> reportlab which survives no sort of reentrancy, for a webservice.
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> <mailto:Twisted-Python at twistedmatrix.com>
> Hi Werner
> I want to know whether you have experienced any serious bug with
> ampoule? What version of ampoule did you use? Is it 0.2.0? Thanks.
Python 2.7.1 Stackless 3.1b3 060516 (release27-maint, Aug 9 2011,
[GCC 4.2.1 20070719 [FreeBSD]] on freebsd8
Type "help", "copyright", "credits" or "license" for more information.
>>> import ampoule
>>> import twisted
sitting in production now for almost six month, a webservice for
template based PDF production with reportlab & z3c.rml.
The only maintenance I do is creating more templates, never done a
restart so far, logs are clean.
As painless as one could wish for
More information about the Twisted-Python