[Twisted-Python] A resizable cooperator class for queuing and dispatching jobs

[Terry Jones]

>>>>> "Glyph" == Glyph Lefkowitz <glyph at twistedmatrix.com> writes:
Glyph> On Dec 8, 2009, at 11:09 PM, Terry Jones wrote:
>> I just wrote a fun class that lets you
>> 
>> - submit jobs to be dispatched to a queue
>> - manage how many tasks are in progress at once
>> - dynamically adjust that number
>> - shut down cleanly, including
>> - recovering jobs that were queued but hadn't been dispatched

Glyph> That does sound like fun!

It *is* fun.

>> # Later... send off some SMS messages.
>> dispatcher.put((2127399921, 'Hello...'))
>> dispatcher.put((5052929919, 'Test...'))

Glyph> Not that this is really germane to the class in question, but, if
Glyph> you're going to be sending SMS messages

That was just an example. I even thought about using strings, but kept it
looking simpler.

>> # Later, bump up to 10 simultaneous jobs.
>> dispatcher.width = 10
>> 
>> # Oops, turns out we're sending too fast, turn it down a little.
>> dispatcher.narrow(3)

Glyph> This seems somewhat asymmetric.  Why are there 'widen' and 'narrow'
Glyph> methods if I can just set the 'width' attribute directly?  I could
Glyph> always just do '.width +=' and '.width -=' if I have a relative
Glyph> value.

Agreed. I started out just implementing widen and narrow, and they (barely)
survived when I later added the width property. I had been thinking it
would be good to just be able to change the width up or down a notch, but
then when I went to write my (a posteriori) tests, I realized that wasn't
really so convenient. So I think you're right they could be simplified
away.

But the code Glyph, the code? Or have I graduated to the point where I no
longer need a laconic garotting?  I think the combo of a DeferredQueue and
a task.Cooperator is pretty deadly.

Terry