[Twisted-Python] running 1,000,000 tasks, 40 at-a-time
terry at jon.es
Wed Oct 26 11:31:40 EDT 2011
> I don't see anything in task.Cooperator to limit the # of
> simultaneously-running tasks. Am I missing something?
I'm not sure, but I think so.
It does limit the number of simultaneously-running tasks.
> I guess, technically, could write my own scheduler that limits the # of
> simultaneously-running tasks. But, then task.Cooperator isn't really
> doing anything useful for me.
I think you need to look at the code more closely (and try using it). The
fact that there's a generator that is yielding deferreds one by one and
that you pass that generator to coop.coiterate the # of simultaneous tasks
that you want has the effect of only launching that many tasks at once and
the cooperator just pulls a new one from the iterator/generator when one of
the previous jobs finishes. It's a bit subtle, but I think it's what you
want. Give the code a try & see :-)
More information about the Twisted-Python