[Twisted-Python] Scalability of an rss-aggregator

Valentino Volonghi aka Dialtone dialtone at aruba.it
Thu Apr 1 05:13:03 EST 2004

Valentino Volonghi aka Dialtone wrote:

> The code locked at 404th feed downloaded without anything running (no 
> parsing and no memoizing).
> No network activity just after the 404th.
> Till the 404th it's not very fast, after the lock (and my Ctrl+C) it 
> goes at light speed till the 758th feed. (and this can be because of
> it's waiting for the feeds to get downloaded, and this happens at 
> about the 400th)

I went on testing and found some interesting things...
I also tried with just 36 feeds and it locked in the same way, then I 
tested again with that feed and everything worked fine.
So maybe it's the resolver that locks.

I tried with the solution, that Andrew posted on the issue tracker but I 
got a lot of this:

[Failure instance: Traceback: twisted.internet.defer.TimeoutError, 
[Query('www.ozzie.net', 255, 1)]
Trying to go on...
[Failure instance: Traceback: exceptions.AttributeError, 'NoneType' 
object has no attribute 'find'
Trying to go on...
Memoizing http://www.ozzie.net/blog/rss.xml ...
finished retrieving
Feed Version: [Failure instance: Traceback: exceptions.AttributeError, 
'NoneType' object has no attribute 'get'
Trying to go on...
Closing connection number 720...

Obviously the error is the first one, since without a feed to parse, all 
the chain fails.

Valentino Volonghi aka Dialtone
Linux User #310274, Gentoo Proud User
X Python Newsreader developer

More information about the Twisted-Python mailing list