[Twisted-Python] A resizable cooperator class for queuing and dispatching jobs

Terry Jones terry at jon.es
Thu Dec 10 04:57:05 EST 2009

>>>>> "Dave" == Dave Peticolas <dave at krondo.com> writes:
Dave> This is really nifty. I know I could use this.

Great. Me too :-)

>> There's also the issue about what to do when the dispatch function hits
>> an error.  An option could be added to re-queue the job, but it's
>> perhaps better to let the dispatch function do that along with whatever
>> else it needs.

Dave> One reason to have a separate error handler is to support generic
Dave> error-handling strategies, like 're-try N times and then send an
Dave> email here', etc. Though maybe you could do that with decorators on
Dave> the dispatch function. It does mean the dispatch function needs to
Dave> know about the task queue, though.

Yes. But I think that's fine, and appropriate. If you're doing something
more sophisticated you can make a class that creates the dispatch queue,
deals with errors, resubmission, etc. That's how I plan to use it.

Dave> Having written something like this, though not as general or as
Dave> elegant, several times, I've found that pause() and resume() is a
Dave> very useful API. That's not the same as setting the width to 0 and
Dave> then back again, as pause() and resume() don't require you to know or
Dave> remember the current width of the queue.

OK, I added pause/resume. You can change the width when it's paused (or on
resume). Jobs can still be added to the queue when the thing is paused,
they just don't get dispatched. The pause method returns a deferred that
fires when all currently underway tasks have completed.  There's also a new
method 'size' which gets you a 2-tuple of the number of tasks currently
underway and the length of the queue. And I made it so you can clear the
queue (perhaps useful if you want to pause, get the pending jobs via
pending(), and clear the queue before resuming, etc).  I also got rid of
narrow/widen, following Glyph's comments.

I'll stick it somewhere more official / permanent soon (I guess into
Launchpad, either in the Fluidinfo PPA or as a sub-project of the tx
collection). For now you can grab the code & the tests at
http://jon.es/other/rdq.tar.bz2 The tests feel funky to me - the ones that
really do anything are all based on expected timings of jobs, queue sizes,
etc. That's what I'm supposed to be testing, but perhaps my way of doing it
is weird.


More information about the Twisted-Python mailing list