[Twisted-Python] Scalability of an rss-aggregator

Valentino Volonghi aka Dialtone dialtone at aruba.it
Thu Apr 1 09:03:07 EST 2004

Andrew Bennetts wrote:

>On Thu, Apr 01, 2004 at 12:13:03PM +0200, Valentino Volonghi aka Dialtone wrote:
>>I went on testing and found some interesting things...
>I just found something interesting too -- I just took a quick peek at
>feedparser, and the parse function looks like it fetches the page
>synchronously using urllib.  That's *not* a good thing to do from inside
>Twisted's main loop.  It looks like you want your parseFeed method to call:
>    r = FeedParser(baseuri)
>    r.feed(data)
>like feedparser.parse does internally.  (You'll need to do a little bit of
>work to return the same sort dictionary that parse constructs for you).
I verified... The parser only downloads if as url is supplied, otherwise 
it does not.

Anyway, since it needs a StringIO like argument, now I do myself the 
conversion into StringIO and this
will make the parser always return as fast as possible from 

I'm starting to think that this is some kind of 'race condition' inside 
twisted, or something similar.
BTW, I'll investigate more and more :)

Valentino Volonghi aka Dialtone
Linux User #310274, Gentoo Proud User
X Python Newsreader developer

More information about the Twisted-Python mailing list