[Twisted-Python] A resizable cooperator class for queuing and dispatching jobs

[Dave Peticolas]

Terry Jones wrote:
>>>>>> "Dave" == Dave Peticolas <dave at krondo.com> writes:
> Dave> This is really nifty. I know I could use this.
> 
> Great. Me too :-)
> 
>>> There's also the issue about what to do when the dispatch function hits
>>> an error.  An option could be added to re-queue the job, but it's
>>> perhaps better to let the dispatch function do that along with whatever
>>> else it needs.
> 
> Dave> One reason to have a separate error handler is to support generic
> Dave> error-handling strategies, like 're-try N times and then send an
> Dave> email here', etc. Though maybe you could do that with decorators on
> Dave> the dispatch function. It does mean the dispatch function needs to
> Dave> know about the task queue, though.
> 
> Yes. But I think that's fine, and appropriate. If you're doing something
> more sophisticated you can make a class that creates the dispatch queue,
> deals with errors, resubmission, etc. That's how I plan to use it.

Makes sense.


> Dave> Having written something like this, though not as general or as
> Dave> elegant, several times, I've found that pause() and resume() is a
> Dave> very useful API. That's not the same as setting the width to 0 and
> Dave> then back again, as pause() and resume() don't require you to know or
> Dave> remember the current width of the queue.
> 
> OK, I added pause/resume. You can change the width when it's paused (or on
> resume). Jobs can still be added to the queue when the thing is paused,
> they just don't get dispatched. The pause method returns a deferred that
> fires when all currently underway tasks have completed.  There's also a new
> method 'size' which gets you a 2-tuple of the number of tasks currently
> underway and the length of the queue. And I made it so you can clear the
> queue (perhaps useful if you want to pause, get the pending jobs via
> pending(), and clear the queue before resuming, etc).  I also got rid of
> narrow/widen, following Glyph's comments.

Cool. What about being able to get the set of underway objects, in
addition to the set of pending objects? Then you can get the whole
state of the queue at that point.

I think:

   queue.resume(0)

doesn't work the way you'd expect at the moment.


> I'll stick it somewhere more official / permanent soon (I guess into
> Launchpad, either in the Fluidinfo PPA or as a sub-project of the tx
> collection). For now you can grab the code & the tests at
> http://jon.es/other/rdq.tar.bz2 The tests feel funky to me - the ones that
> really do anything are all based on expected timings of jobs, queue sizes,
> etc. That's what I'm supposed to be testing, but perhaps my way of doing it
> is weird.
> 
> Terry
>