[Twisted-Python] Persistent Queue with subscriber worker
esteve at sindominio.net
Mon Nov 3 07:32:50 EST 2008
On Monday 03 November 2008 09:40:14 Daniel de la Cuesta wrote:
> I want to develop a video conversion service. My idea is to offer a REST
> api and save all the conversion request in a database to track the state of
> each process (pending, doing, done, error, etc.)
Do you plan to expose this REST api using Twisted?
> I want to use this database table as a persistent queue and a twisted
> daemon that takes its task from this table. This daemon has to take the
> task, update the state of the task and do the work.
> How can I subscribe the twisted daemon to the database inserts?
> Is the thread pool a good approach to this problem?
> Do you know a more simple solution to this problem?
Well, I wouldn't use a database at all (except for logging purposes only
maybe). Actually you're already using the proper terms (queue, subscriber,
etc.), so I would use a message-driven architecture:
- A messaging broker, either using AMQP (RabbitMQ, Qpid and OpenAMQ) or Stomp
- You have a subscriber (waiting for messages on a queue), which dispatches
tasks from the queue
- The web application sends messages to the messaging broker, which will route
them to the right subscriber
One of the advantages of this scenario is that you can add more subscribers at
runtime if needed, which will distribute the load between several conversion
You can choose either AMQP  or Stomp , each has its strengths and
1 - https://launchpad.net/txamqp (disclaimer, I'm one of the developers)
2 - http://code.google.com/p/stomper/
More information about the Twisted-Python