[Twisted-Python] Scalability of an rss-aggregator

[Valentino Volonghi aka Dialtone]

Valentino Volonghi aka Dialtone wrote:

> Maybe the problem is there, but then I wouldn't answer the other 
> question:
> "Why does it takes at most 30 second to parse all the remaining 350 
> feeds?"
> There is no network activity after the unlocking "Ctrl+C"...
> Gotta investigate then.

Now... The problem is not a parsing problem.
I just made a last test (and am performing another one right now with 
spewer).

The code locked at 404th feed downloaded without anything running (no 
parsing and no memoizing).

No network activity just after the 404th.
Till the 404th it's not very fast, after the lock (and my Ctrl+C) it 
goes at light speed till the 758th feed. (and this can be because of
it's waiting for the feeds to get downloaded, and this happens at about 
the 400th)

Ok, now the "debug" version has just stopped, here is the last output:

function callWithLogger in 
/usr/lib/python2.3/site-packages/twisted/python/log.py, line 54
method logPrefix of twisted.internet.tcp.Client at 1085149132
function callWithContext in 
/usr/lib/python2.3/site-packages/twisted/python/log.py, line 49
method getContext of twisted.python.context.ContextTracker at 1081585292
method callWithContext of twisted.python.context.ContextTracker at 
1081585292
method _doReadOrWrite of twisted.internet.default.SelectReactor at 
1077521228
method doRead of twisted.internet.tcp.Client at 1085149132
method fileno of socket._socketobject at 1085157164
method removeReader of twisted.internet.default.SelectReactor at 1077521228
method removeWriter of twisted.internet.default.SelectReactor at 1077521228
method connectionLost of twisted.internet.tcp.Client at 1085149132
method connectionLost of twisted.internet.tcp.Client at 1085149132
method connectionLost of twisted.internet.tcp.Client at 1085149132
method _closeSocket of twisted.internet.tcp.Client at 1085149132
method shutdown of socket._socketobject at 1085157164
method connectionLost of twisted.web.client.HTTPPageGetter at 1086481388
method connectionLost of twisted.web.client.HTTPPageGetter at 1086481388
method handleResponseEnd of twisted.web.client.HTTPPageGetter at 1086481388
method noPage of twisted.web.client.HTTPClientFactory at 1085148876
method connectionLost of twisted.internet.tcp.Connector at 1085149100
method clientConnectionLost of twisted.web.client.HTTPClientFactory at 
1085148876
method doStop of twisted.web.client.HTTPClientFactory at 1085148876
method __repr__ of twisted.web.client.HTTPClientFactory at 1085148876
method msg of twisted.python.log.LogPublisher at 1081585612
method getContext of twisted.python.context.ContextTracker at 1081585292
method _emit of twisted.python.log.DefaultObserver at 1081585644
method stopFactory of twisted.web.client.HTTPClientFactory at 1085148876
method runUntilCurrent of twisted.internet.default.SelectReactor at 
1077521228
method timeout of twisted.internet.default.SelectReactor at 1077521228
method doSelect of twisted.internet.default.SelectReactor at 1077521228
method fileno of socket._socketobject at 1086221044
method fileno of socket._socketobject at 1086589444
method fileno of socket._socketobject at 1085552996
method fileno of socket._socketobject at 1086453556
[LOTS OF THESE]
method fileno of socket._socketobject at 1086477372
method fileno of socket._socketobject at 1086623612
method fileno of socket._socketobject at 1085203316
method fileno of socket._socketobject at 1085733820

##@#@#@ Ctrl+C

And now it goes on till the end at warp speed.

-- 
Valentino Volonghi aka Dialtone
Linux User #310274, Gentoo Proud User
X Python Newsreader developer
http://sourceforge.net/projects/xpn/