[Twisted-Python] I could swear I've seen this pattern *somewhere* in Twisted...

[Mike C. Fletcher]

I have a little utility module "parallel", that dispatches a callable
for "count" items in parallel, collecting them as would a DeferredList
(attached for reference).  The idea here is that you want only count
items to be actively processing at any given time, and you want to be
notified when the entire suite of items completes processing (just like
a DeferredList).

I use this when I have, for instance, 10,000 items to dispatch as part
of a "batch" process, but I don't want to negatively impact the
responsiveness of the user's front-end to the system by adding 10,000
active operations to the internal Twisted queues.  The *correct*
solution for these problems would probably be to have a
priority-handling mechanism, to allow those 10,000 items not to
interfere with high-priority events, but until we have that, it's
necessary to do these kinds of things to keep the system responsive with
large batch operations occurring in the background.

My question being (yay, finally got around to it), where is this pattern
in Twisted.  As I said in the subject, I think I've seen *something*
described as implementing this pattern, and I'd rather use a standard
mechanism than something I've constructed myself.

Enjoy all,
Mike

-- 
________________________________________________
  Mike C. Fletcher
  Designer, VR Plumber, Coder
  http://www.vrplumber.com
  http://blog.vrplumber.com

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: parallel.py
URL: </pipermail/twisted-python/attachments/20050509/af028d2e/attachment.ksh>