[Twisted-Python] Re: Twisted and the Posh Module
general at eepatents.com
Mon Mar 14 13:59:51 EST 2005
On Monday 14 March 2005 10:13 am,
twisted-python-request at twistedmatrix.com wrote:
> Has anyone tried playing with Twisted and the posh module? I need to
> do some CPU-intensive stuff inside the reactor, which unfortunately
> holds on to the GIL. I was thinking of just making a deferToPosh()
> type of method in the spirit of deferToThread(), but forking on each
> operation would be pretty expensive. Which brings up the question of
> having process pools.
> Has anyone done anything like this? I don't really need any
> interaction between the worker processes and Twisted, so in theory I
> could spawn off some worker processes early and just communicate
> queues of tasks to complete, independent of Twisted and the reactor.
I think having some sort of process pool mangagement in Twisted is a
great idea, especially with multi-core CPUs emerging on the scene. I
have access to a dual-core Pentium Prescott CPU and it would be great
to have that available to keep both cores humming on a certain
CPU-intensive project I'm considering.
However, I'm not sure the best way to go about it would be with the posh
module, pypar (see http://datamining.anu.edu.au/~ole/pypar/), or just
using Perspective Broker as an underlying message-passing mechanism
with UNIX sockets and/or TCP. One thought might be to have a single
master process start up and act as a PB server and process pool
manager. Subsidiary processes could then make authenticated PB
connections to the server to "volunteer" for work in the process pool.
Note that pypar lets you easily find out how many CPUs you have under
kernel control, with pypar.size(). Thus, the main process could start
the process pool by spawning a subsidiary "volunteer" process for each
CPU core present.
Registered Patent Agent
Open-Source Software Author (yes, both...)
Web Site: http://www.eepatents.com
More information about the Twisted-Python