[Twisted-Python] Ideas on limiting/throttling spawnProcess

[Nicolas Toper]

For a similar use case, we're using a combination of cooperator with
defertoprocess.

I can explain more if anyone's interested.
Le 29 oct. 2011 00:41, "Donal McMullan" <donal.mcmullan at gmail.com> a écrit :

> I think twisted.internet.defer.DeferredSemaphore is designed for this. John
> Paul Calderone answers a similar question in detail:
>
> http://stackoverflow.com/questions/2861858/queue-remote-calls-to-a-python-twisted-perspective-broker
>
> Donal McMullan
>
>
> On 29/10/2011, at 11:21 AM, Daryl Herzmann wrote:
>
> > Hi,
> >
> > I have an application that ingests data and does a reactor.spawnProcess()
> > for each chunk of data (a product) to pass it as STDIN to a compiled
> binary
> > and then I harvest the STDOUT. It has been working well, but I have an
> issue
> > when my data rates get too high and the machine gets overloaded with
> spawned
> > processes (I think) and starts running out of file descriptors (yes I can
> > raise it :) causing all sorts of pain.  I'm wondering about a mechanism
> to
> > throttle the number of spawned processes going at one time?  It'd be nice
> to
> > only have 10 of these spawned processes going at any one time.  Thanks
> for
> > your ideas :)
> >
> > daryl
> >
> > code snipet:
> >
> > class SHEFIT(protocol.ProcessProtocol):
> >    def __init__(self, tp):
> >        self.tp = tp
> >        self.data = ""
> >
> >    def connectionMade(self):
> >        self.transport.write( self.tp.raw )
> >        self.transport.closeStdin()
> >
> >    def outReceived(self, data):
> >        self.data = self.data + data
> >
> >    def errReceived(self, data):
> >        print "errReceived! with %d bytes!" % len(data)
> >        print data
> >
> >    def outConnectionLost(self):
> >        really_process(self.tp, self.data)
> >
> > def got_product():
> >    shef = SHEFIT( tp )
> >    reactor.spawnProcess(shef, "shefit", ["shefit"], {})
> >
> > def really_process(tp,data):
> >    print 'Do some work'
> >
> > _______________________________________________
> > Twisted-Python mailing list
> > Twisted-Python at twistedmatrix.com
> > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20111029/930c1428/attachment-0002.html>