[Twisted-Python] Ideas on limiting/throttling spawnProcess

[Nicolas Toper]

def parallel(iterable, count, callable, *args, **named):
    #Copy/paste from http://jcalderone.livejournal.com/24285.html
    #
http://oubiwann.blogspot.com/2008/06/async-batching-with-twisted-walkthrough.html
    coop = task.Cooperator()
    work = (callable(elem, *args, **named) for elem in iterable)
    return defer.DeferredList([coop.coiterate(work) for i in xrange(count)])


def _localSend(messages):
    return deferToProcessPool(reactor, _p, makeEmails, messages, logger)

_p = Pool(5)

def _localDeliver(messages):
   #... Some part left out
    d = parallel(messages,2, _localSend)
    d.addErrback(_log_error)
    return d

_localDeliver call parallel: it will launch at most 2 deferred executing
localSend(messages). These are actually deferToProcess

This is a very very powerful compbination and exarkun really nailed it in
his blog :)


On Mon, Oct 31, 2011 at 2:22 PM, Daryl Herzmann <akrherz at iastate.edu> wrote:

> On Sat, Oct 29, 2011 at 2:10 AM, Nicolas Toper <ntoper at gmail.com> wrote:
> > For a similar use case, we're using a combination of cooperator with
> > defertoprocess.
> >
> > I can explain more if anyone's interested.
>
> I would certainly be interested :)  I'm having a difficult time
> understanding how the pieces fit together.  Thank you!
>
> daryl
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20111031/0450503f/attachment.htm