[Twisted-Python] Persistent Queue with subscriber worker

[Esteve Fernandez]

Hi Daniel

On Monday 03 November 2008 09:40:14 Daniel de la Cuesta wrote:
>  I want to develop a video conversion service. My idea is to offer a REST
> api and save all the conversion request in a database to track the state of
> each process (pending, doing, done, error, etc.)

Do you plan to expose this REST api using Twisted?

>  I want to use this database table as a persistent queue and a twisted
> daemon that takes its task from this table. This daemon has to take the
> task, update the state of the task and do the work.
>
>  How can I subscribe the twisted daemon to the database inserts?
>
>  Is the thread pool a good approach to this problem?
>
>  Do you know a more simple solution to this problem?

Well, I wouldn't use a database at all (except for logging purposes only 
maybe). Actually you're already using the proper terms (queue, subscriber, 
etc.), so I would use a message-driven architecture:

- A messaging broker, either using AMQP (RabbitMQ, Qpid and OpenAMQ) or Stomp 
(RabbitMQ, ActiveMQ)
- You have a subscriber (waiting for messages on a queue), which dispatches 
tasks from the queue
- The web application sends messages to the messaging broker, which will route 
them to the right subscriber

One of the advantages of this scenario is that you can add more subscribers at 
runtime if needed, which will distribute the load between several conversion 
servers.

You can choose either AMQP [1] or Stomp [2], each has its strengths and 
weaknesses.

Cheers.

1 - https://launchpad.net/txamqp (disclaimer, I'm one of the developers)
2 - http://code.google.com/p/stomper/