[Twisted-web] Twisted.web/Twisted.web2 and multiple threads

Phil Mayers p.mayers at imperial.ac.uk
Fri Oct 12 07:37:57 EDT 2007


On Fri, 2007-10-12 at 00:34 +0200, Fabien Schwob wrote:
> Hello,
> 
> I'm new to twisted and I would like to know if the following problems 
> can be solved with Twisted.
> 
> My goal is to create a daemon that will download RSS/Atom feeds, parse 
> them and insert various information in the database. So my idea is to 
> use twisted create a program that will have two "main" threads :

You don't use threads with twisted. Twisted is an async event-driven
framework.

The only use for threads in twisted is calling non-Twisted APIs that
block (and ideally release the python GIL). The most prominent instance
of this is the twisted.enterprise.adbapi wrapper around the python
dbapi.

> 
> * the first on is an HTTP REST interface to allow the user to add feeds 
> url in a queue. I think I will use twisted.web ou web2 here
> 
> * the second thread launch several child thread that download feeds and 
> parse the. I take feeds from the queue and parse them.

You don't do it that way (though possibly you might want to run the XML
parsing in a thread if the feeds are large).

I suggest you look at the Twisted examples to get a feel for running a
server and a client in the same thread.


> 
> So the goal is to transform the http://dpaste.com/22198/ script into a 
> daemon where I can add feed with a REST API. So, what is the base 
> solutions to do that with twisted.
> 
> Thanks
> 




More information about the Twisted-web mailing list