[Twisted-Python] Ideas on limiting/throttling spawnProcess
ntoper at gmail.com
Mon Oct 31 16:43:49 EDT 2011
def parallel(iterable, count, callable, *args, **named):
#Copy/paste from http://jcalderone.livejournal.com/24285.html
coop = task.Cooperator()
work = (callable(elem, *args, **named) for elem in iterable)
return defer.DeferredList([coop.coiterate(work) for i in xrange(count)])
return deferToProcessPool(reactor, _p, makeEmails, messages, logger)
_p = Pool(5)
#... Some part left out
d = parallel(messages,2, _localSend)
_localDeliver call parallel: it will launch at most 2 deferred executing
localSend(messages). These are actually deferToProcess
This is a very very powerful compbination and exarkun really nailed it in
his blog :)
On Mon, Oct 31, 2011 at 2:22 PM, Daryl Herzmann <akrherz at iastate.edu> wrote:
> On Sat, Oct 29, 2011 at 2:10 AM, Nicolas Toper <ntoper at gmail.com> wrote:
> > For a similar use case, we're using a combination of cooperator with
> > defertoprocess.
> > I can explain more if anyone's interested.
> I would certainly be interested :) I'm having a difficult time
> understanding how the pieces fit together. Thank you!
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Twisted-Python