[Twisted-Python] Ideas on limiting/throttling spawnProcess

Valentino Volonghi dialtone at gmail.com
Mon Oct 31 17:49:38 EDT 2011

On 28 Oct 2011, at 15:21, Daryl Herzmann wrote:

> Hi,
> I have an application that ingests data and does a reactor.spawnProcess()
> for each chunk of data (a product) to pass it as STDIN to a compiled binary
> and then I harvest the STDOUT. It has been working well, but I have an issue
> when my data rates get too high and the machine gets overloaded with spawned
> processes (I think) and starts running out of file descriptors (yes I can
> raise it :) causing all sorts of pain.  I'm wondering about a mechanism to
> throttle the number of spawned processes going at one time?  It'd be nice to
> only have 10 of these spawned processes going at any one time.  Thanks for
> your ideas :)

Another alternative is to use turtl which was written for a similar purpose.


Valentino Volonghi

More information about the Twisted-Python mailing list