[Twisted-Python] Scalability of an rss-aggregator

[Valentino Volonghi aka Dialtone]

Valentino Volonghi aka Dialtone wrote:

> The code locked at 404th feed downloaded without anything running (no 
> parsing and no memoizing).
>
> No network activity just after the 404th.
> Till the 404th it's not very fast, after the lock (and my Ctrl+C) it 
> goes at light speed till the 758th feed. (and this can be because of
> it's waiting for the feeds to get downloaded, and this happens at 
> about the 400th)

I went on testing and found some interesting things...
I also tried with just 36 feeds and it locked in the same way, then I 
tested again with that feed and everything worked fine.
So maybe it's the resolver that locks.

I tried with the solution, that Andrew posted on the issue tracker but I 
got a lot of this:

[Failure instance: Traceback: twisted.internet.defer.TimeoutError, 
[Query('www.ozzie.net', 255, 1)]
]
====================
Trying to go on...
parsing...
[Failure instance: Traceback: exceptions.AttributeError, 'NoneType' 
object has no attribute 'find'
/usr/lib/python2.3/site-packages/twisted/internet/defer.py:313:_runCallbacks
twisted-rss.py:107:parseFeed
/home/dialtone/programmi_didattici/rss-aggregator/feedparser.py:1679:parse
/home/dialtone/programmi_didattici/rss-aggregator/feedparser.py:1289:_open_resource
/usr/lib/python2.3/urlparse.py:49:urlparse
/usr/lib/python2.3/urlparse.py:79:urlsplit
]
====================
Trying to go on...
Memoizing http://www.ozzie.net/blog/rss.xml ...
--------------------
finished retrieving
Feed Version: [Failure instance: Traceback: exceptions.AttributeError, 
'NoneType' object has no attribute 'get'
/usr/lib/python2.3/site-packages/twisted/internet/defer.py:313:_runCallbacks
twisted-rss.py:123:workOnPage
]
====================
Trying to go on...
Closing connection number 720...
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Obviously the error is the first one, since without a feed to parse, all 
the chain fails.

-- 
Valentino Volonghi aka Dialtone
Linux User #310274, Gentoo Proud User
X Python Newsreader developer
http://sourceforge.net/projects/xpn/