[Twisted-Python] feeding-worker-loop problem

[Patrick Scharrenberg]

Hi!

I've been thinking for some time now about the following problem, but
can't find a solution.

I have a queue with a fluctuant number of tasks. This queue might be
empty at some time but might also become such full, that all workers
have work to do and we have to wait until one becomes free again.

I'd like to have a number of workers which process these tasks from the
queue.

What I've got:
- the worker, which returns a deferred for further processing
- a worker-pool class, which hands out the workers with an attached
callback to re-queue them to the "available-pool"

My problem:
- the connection between the queue and the worker-pool and integrating
this into the main reactor-loop

My first thought was a twisted.internet.task.LoopingCall, trying to get
a worker and on success giving it a task, but that's not very
asynchronous, isn't it? Especially no when the queue is empty.

I've no clue how to implement this.

I've read something about producer-consumer in twisted, but the
documentation here is very rare and I think it's not what I'm looking for.


Can someone please give me a hint, if twisted offers some techniques to
implement this, or point me in the right direction?


Thanks
Patrick