[Twisted-Python] running 1,000,000 tasks, 40 at-a-time

[Jason Rennie]

On Wed, Oct 26, 2011 at 10:22 AM, Terry Jones <terry at jon.es> wrote:

> Sounds like you should be using a Python generator and that you're not.
>

The issue I'm seeing is that the ~million DeferredSemaphore.run() calls are
inefficient; it's independent of list/generator.

First off, have you read http://jcalderone.livejournal.com/24285.html ?
> If not, do.  Take note of the line
>
>  work = (callable(elem, *args, **named) for elem in iterable)
>
> work is a generator.  Make sure you understand every line of that code :-)
>

I don't see anything in task.Cooperator to limit the # of
simultaneously-running tasks.  Am I missing something?  I guess,
technically, could write my own scheduler that limits the # of
simultaneously-running tasks.  But, then task.Cooperator isn't really doing
anything useful for me.

You might also find https://launchpad.net/txrdq of interest, though it's
> probably overkill for what you're trying to do.
>

Yes, it is overkill.

Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20111026/3d51a65f/attachment-0001.htm