[Twisted-Python] AsynQueue: return deferred from queued method

[Ed Suominen]

Stephan Jaeger wrote:
> I'm trying to use AsynQueue for queuing requests for a synchronous
> protocol (emi/ucp). Is it possible to return a deferred from the queued
> method so the next method in the queue is only called after the deferred
> from the first was fired? Or is there a better way to handle synchronous
> protocols where i have to wait for an answer of a message sent over the
> wire before i can send another one?

Just use a single implementer of IWorker that only accepts one 
assignment at a time. The queue will not feed it its next task until the 
deferred from its previous one has fired.

What you're trying to do is exactly what AsynQueue is for. For example, 
my sAsync package uses AsynQueue to feed database queries to the SQLite 
database engine one at a time in a single thread.

If you have multiple connections that each can handle successive 
requests successively, but independently the other connection, you can 
simply use an IWorker implementer for each one. (I don't know that 
emi/ucp has any way of doing this.) Then the queue will feed requests to 
whoever is ready, one at a time.

I use this for running jobs asynchronously over a cluster of CPU nodes. 
The queue gets fed thousands of tasks nearly at once, and parcels them 
out one at a time to each node. Some CPU nodes run faster than others, 
and the queueing makes that all work out fine.

Please note that there is also a very simple and elegant queue object in 
twisted.internet.defer that might meet your needs without the complexity 
(and capability) of AsynQueue.

Feel free to contact me offline if you have additional questions on this.

Best regards,
Ed Suominen