[Twisted-Python] Understanding Deferreds/callback further

Samuel Reynolds sam at spinwardstars.com
Wed Nov 8 23:42:15 EST 2006

>Ok.  The thing I am trying to design/code is a job distribution 
>system, for a distributed computation model.  There is a server 
>which interacts with the job database (which is just a bsddb).  The 
>server accepts connections from clients and distributes jobs to 
>them.  The server also accepts connections from a 2nd type of 
>client, which submits jobs to the server.

Sounds like a distributed task queue.

Users/user processes submit tasks to the master queue
manager (the "server") via a defined interface (HTTP post,
SOAP, XML-RPC, PB, or ???). The master queue manager
adds the requested tasks to the master job queue.

A secondary queue manager on each "client" machine
(the "monitor") requests tasks from the master queue
when its local queue drops below a defined threshold
number of jobs (or drops to zero). When the local queue
is empty, the secondary queue manager periodically
requests tasks from the master. It adds any tasks
received to its local queue.

For each received task, the secondary queue manager
starts a worker task, immediately receiving a deferred.
It registers success and error callbacks on the deferred.

When it gets the success or error callback from a worker,
the secondary queue manager records the result and reports
it to the master queue controller (the "server"). When it
has successfully reported to the master, it deletes the
task from the local queue.

Additional wrinkles: (1) Intertask dependencies may require
that some tasks be performed sequentially, on the same
slave machine, or both. (2) Some tasks may be more important
than others, and so must be processed (or at least started)
first. (3) Task cancellation at the master may require
that the master be able to send asynchronous task-cancel
instructions to the slaves.

>The workers of course have to submit their results back to the 
>server, but in the system I envisioned the monitor takes care of 
>that by continuously polling the status of the workers.

If the worker returns a deferred, the monitor doesn't
need to poll; it'll get a callback when the worker
completes (or fails).


- Sam 

More information about the Twisted-Python mailing list