[Twisted-Python] Persistent Queue with subscriber worker

[Phil Mayers]

Daniel de la Cuesta wrote:
> Hi,
> 
> I want to develop a video conversion service. My idea is to offer a REST 
> api and save all the conversion request in a database to track the state 
> of each process (pending, doing, done, error, etc.)
> 
> I want to use this database table as a persistent queue and a twisted 
> daemon that takes its task from this table. This daemon has to take the 
> task, update the state of the task and do the work.
> 
> How can I subscribe the twisted daemon to the database inserts?

The easiest way is to use twisted.internet.task.LoopingCall and check 
for new entries every N seconds (e.g. n=1, n=30)

If your database supports things like postgres' NOTIFY and you have an 
async-aware db adapter (e.g. pgasync) you could use a trigger on the 
table to issue a NOTIFY and respond immediately, but it's been my 
experience that's more trouble than it's worth.

> 
> Is the thread pool a good approach to this problem?

Since most DB adapters are blocking you'll want to use something like a 
thread pool to access them - twisted's adbapi module does this for you.