[Twisted-Python] running 1,000,000 tasks, 40 at-a-time

[Terry Jones]

Hi Jason

> I don't see anything in task.Cooperator to limit the # of
> simultaneously-running tasks.  Am I missing something?

I'm not sure, but I think so.

It does limit the number of simultaneously-running tasks.

> I guess, technically, could write my own scheduler that limits the # of
> simultaneously-running tasks.  But, then task.Cooperator isn't really
> doing anything useful for me.

I think you need to look at the code more closely (and try using it). The
fact that there's a generator that is yielding deferreds one by one and
that you pass that generator to coop.coiterate the # of simultaneous tasks
that you want has the effect of only launching that many tasks at once and
the cooperator just pulls a new one from the iterator/generator when one of
the previous jobs finishes. It's a bit subtle, but I think it's what you
want. Give the code a try & see :-)

T