[Twisted-Python] Persistent Queue with subscriber worker
p.mayers at imperial.ac.uk
Mon Nov 3 05:31:57 EST 2008
Daniel de la Cuesta wrote:
> I want to develop a video conversion service. My idea is to offer a REST
> api and save all the conversion request in a database to track the state
> of each process (pending, doing, done, error, etc.)
> I want to use this database table as a persistent queue and a twisted
> daemon that takes its task from this table. This daemon has to take the
> task, update the state of the task and do the work.
> How can I subscribe the twisted daemon to the database inserts?
The easiest way is to use twisted.internet.task.LoopingCall and check
for new entries every N seconds (e.g. n=1, n=30)
If your database supports things like postgres' NOTIFY and you have an
async-aware db adapter (e.g. pgasync) you could use a trigger on the
table to issue a NOTIFY and respond immediately, but it's been my
experience that's more trouble than it's worth.
> Is the thread pool a good approach to this problem?
Since most DB adapters are blocking you'll want to use something like a
thread pool to access them - twisted's adbapi module does this for you.
More information about the Twisted-Python