<div>def parallel(iterable, count, callable, *args, **named):</div><div> #Copy/paste from <a href="http://jcalderone.livejournal.com/24285.html">http://jcalderone.livejournal.com/24285.html</a></div><div> #<a href="http://oubiwann.blogspot.com/2008/06/async-batching-with-twisted-walkthrough.html">http://oubiwann.blogspot.com/2008/06/async-batching-with-twisted-walkthrough.html</a></div>
<div> coop = task.Cooperator()</div><div> work = (callable(elem, *args, **named) for elem in iterable)</div><div> return defer.DeferredList([coop.coiterate(work) for i in xrange(count)])</div><div><br></div><div>
<br></div><div>def _localSend(messages):</div><div> return deferToProcessPool(reactor, _p, makeEmails, messages, logger)</div><div><br></div><div>_p = Pool(5)</div><div><br></div><div><div>def _localDeliver(messages):</div>
<div> #... Some part left out</div><div> d = parallel(messages,2, _localSend)</div><div> d.addErrback(_log_error)</div><div> return d</div></div><div><br></div><div>_localDeliver call parallel: it will launch at most 2 deferred executing localSend(messages). These are actually deferToProcess</div>
<div><br></div><div>This is a very very powerful compbination and exarkun really nailed it in his blog :)</div><br><br><div class="gmail_quote">On Mon, Oct 31, 2011 at 2:22 PM, Daryl Herzmann <span dir="ltr"><<a href="mailto:akrherz@iastate.edu">akrherz@iastate.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im">On Sat, Oct 29, 2011 at 2:10 AM, Nicolas Toper <<a href="mailto:ntoper@gmail.com">ntoper@gmail.com</a>> wrote:<br>
> For a similar use case, we're using a combination of cooperator with<br>
> defertoprocess.<br>
><br>
> I can explain more if anyone's interested.<br>
<br>
</div>I would certainly be interested :) I'm having a difficult time<br>
understanding how the pieces fit together. Thank you!<br>
<font color="#888888"><br>
daryl<br>
</font><div><div></div><div class="h5"><br>
_______________________________________________<br>
Twisted-Python mailing list<br>
<a href="mailto:Twisted-Python@twistedmatrix.com">Twisted-Python@twistedmatrix.com</a><br>
<a href="http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python" target="_blank">http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python</a><br>
</div></div></blockquote></div><br>