[Twisted-Python] Ideas on limiting/throttling spawnProcess

[Donal McMullan]

I think twisted.internet.defer.DeferredSemaphore is designed for this. John Paul Calderone answers a similar question in detail:
http://stackoverflow.com/questions/2861858/queue-remote-calls-to-a-python-twisted-perspective-broker

Donal McMullan


On 29/10/2011, at 11:21 AM, Daryl Herzmann wrote:

> Hi,
> 
> I have an application that ingests data and does a reactor.spawnProcess()
> for each chunk of data (a product) to pass it as STDIN to a compiled binary
> and then I harvest the STDOUT. It has been working well, but I have an issue
> when my data rates get too high and the machine gets overloaded with spawned
> processes (I think) and starts running out of file descriptors (yes I can
> raise it :) causing all sorts of pain.  I'm wondering about a mechanism to
> throttle the number of spawned processes going at one time?  It'd be nice to
> only have 10 of these spawned processes going at any one time.  Thanks for
> your ideas :)
> 
> daryl
> 
> code snipet:
> 
> class SHEFIT(protocol.ProcessProtocol):
>    def __init__(self, tp):
>        self.tp = tp
>        self.data = ""
> 
>    def connectionMade(self):
>        self.transport.write( self.tp.raw )
>        self.transport.closeStdin()
> 
>    def outReceived(self, data):
>        self.data = self.data + data
> 
>    def errReceived(self, data):
>        print "errReceived! with %d bytes!" % len(data)
>        print data
> 
>    def outConnectionLost(self):
>        really_process(self.tp, self.data)
> 
> def got_product():
>    shef = SHEFIT( tp )
>    reactor.spawnProcess(shef, "shefit", ["shefit"], {})
> 
> def really_process(tp,data):
>    print 'Do some work'
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python