[Twisted-Python] scaling with lots of clients
platsakos at gmail.com
Wed Aug 3 20:04:14 EDT 2011
Problems might arise if you use multiple processes, each with each own
reactor. Speaking from (limited, admittedly experience), using say two
reactors in two processes respectively, if there is a need to
communicate between them might lead to all kinds of bugs, *if* you do
not do things properly.
Another workaround would be the following:
Python's GIL lock applies only to executing Python bytecode. Therefore,
if you offload some of it to a C-based module, you are GIL-lock free.
On 08/04/2011 01:42 AM, Grant Mckenzie wrote:
>> Threading won't help much if you're CPU bound, since Python can only run
>> one Python thread at once. Process pooling probably would help, though
>> will involve a more complex infrastructure. Which part of the docs made
>> you think that process pooling is a bad idea?
> Hi Itamar,
> on re-reading I see nothing particular about process pooling being a bad idea.
> I see the explanation of spawning processes from the reactor here:
> This seems to be designed for spawning processes for compute purposes which no doubt has a lot of use cases. I'd like ot spawn
> processes to handle user communication however. THe only thing that I can think of is multiple processes each with their
> own reactor handling communication, and one or more other processes to handle compute. Thoughts?
> THanks for your help.
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
What is the air-speed velocity of an unladen swallow?
More information about the Twisted-Python