<p>For a similar use case, we're using a combination of cooperator with defertoprocess.</p>
<p>I can explain more if anyone's interested. </p>
<div class="gmail_quote">Le 29 oct. 2011 00:41, "Donal McMullan" <<a href="mailto:donal.mcmullan@gmail.com">donal.mcmullan@gmail.com</a>> a écrit :<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I think twisted.internet.defer.DeferredSemaphore is designed for this. John Paul Calderone answers a similar question in detail:<br>
<a href="http://stackoverflow.com/questions/2861858/queue-remote-calls-to-a-python-twisted-perspective-broker" target="_blank">http://stackoverflow.com/questions/2861858/queue-remote-calls-to-a-python-twisted-perspective-broker</a><br>
<br>
Donal McMullan<br>
<br>
<br>
On 29/10/2011, at 11:21 AM, Daryl Herzmann wrote:<br>
<br>
> Hi,<br>
><br>
> I have an application that ingests data and does a reactor.spawnProcess()<br>
> for each chunk of data (a product) to pass it as STDIN to a compiled binary<br>
> and then I harvest the STDOUT. It has been working well, but I have an issue<br>
> when my data rates get too high and the machine gets overloaded with spawned<br>
> processes (I think) and starts running out of file descriptors (yes I can<br>
> raise it :) causing all sorts of pain. I'm wondering about a mechanism to<br>
> throttle the number of spawned processes going at one time? It'd be nice to<br>
> only have 10 of these spawned processes going at any one time. Thanks for<br>
> your ideas :)<br>
><br>
> daryl<br>
><br>
> code snipet:<br>
><br>
> class SHEFIT(protocol.ProcessProtocol):<br>
> def __init__(self, tp):<br>
> <a href="http://self.tp" target="_blank">self.tp</a> = tp<br>
> self.data = ""<br>
><br>
> def connectionMade(self):<br>
> self.transport.write( self.tp.raw )<br>
> self.transport.closeStdin()<br>
><br>
> def outReceived(self, data):<br>
> self.data = self.data + data<br>
><br>
> def errReceived(self, data):<br>
> print "errReceived! with %d bytes!" % len(data)<br>
> print data<br>
><br>
> def outConnectionLost(self):<br>
> really_process(<a href="http://self.tp" target="_blank">self.tp</a>, self.data)<br>
><br>
> def got_product():<br>
> shef = SHEFIT( tp )<br>
> reactor.spawnProcess(shef, "shefit", ["shefit"], {})<br>
><br>
> def really_process(tp,data):<br>
> print 'Do some work'<br>
><br>
> _______________________________________________<br>
> Twisted-Python mailing list<br>
> <a href="mailto:Twisted-Python@twistedmatrix.com">Twisted-Python@twistedmatrix.com</a><br>
> <a href="http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python" target="_blank">http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python</a><br>
<br>
<br>
_______________________________________________<br>
Twisted-Python mailing list<br>
<a href="mailto:Twisted-Python@twistedmatrix.com">Twisted-Python@twistedmatrix.com</a><br>
<a href="http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python" target="_blank">http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python</a><br>
</blockquote></div>