Your limit will usually be the number of file descriptors in the system, which can be usually changed via ulimit or your system's equivalent. On Linux I believe it defaults to 1024, so you should be able to handle 1024 simultaneous connections.<br>
<br>One thing of note is that you say you have concurrency issues handled -- but with asynchronous I/O, there are no concurrency issues, since there's no concurrency (at least, not at application level). This is confusing at first but it's important to understand.<br>
<br>All that said, you probably want to maintain a queue of URLs and some sort of graph representation of your data for purposes of finding loops (e.g. A links to B, B links to C, C links to A). You can then set an upper limit on the number of concurrent connections (say 1000) and track the number of deferreds in the system just based on when you start connections and when they finish (via callbacks). Your initial seed can start one URL, and then its callback can hit all linked nodes, and so on and so on.<br>
<br>You might be hitting a cycle in the page traversal graph, and that is causing you all sorts of problems in terms of recursion depth or running out of file descriptors. Without seeing your code or your target site, though, it's impossible to say.<br>
<br>Have you considered using another library for web spidering? I believe Scrapy (<a href="http://scrapy.org">http://scrapy.org</a>) is a good spidering tool, and it might be easier to use a decent library than roll your own.<br>
<br><br> - Matt<br><br><br><br><div class="gmail_quote">On Tue, Oct 6, 2009 at 10:40 PM, Steve Steiner (listsin) <span dir="ltr"><<a href="mailto:listsin@integrateddevcorp.com">listsin@integrateddevcorp.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">So, I have a situation...<br>
<br>
I have an application whose basic function is, in simplified form:<br>
<br>
def main():<br>
get_web_page(main_page_from_params)<br>
<br>
def get_web_page(page_name):<br>
set up a page getter deferred,<br>
one of the callbacks gets the links out of the page and sends them<br>
to get_them()<br>
<br>
def get_them(links):<br>
for l in links:<br>
if l is not being gotten or hasn't been got:<br>
deferred = get_web_page(l)<br>
<br>
In other words, I feed in the top level page, then recursively feed<br>
in any pages linked to by the current page, and they feed in all their<br>
links, until all pages are gotten.<br>
<br>
I understand the concurrency issues with multiple deferred's trying<br>
to add pages to the "get list" -- it's properly handled in the code<br>
(far as I can tell, so far).<br>
<br>
So, here's the question...<br>
<br>
I have a "pages" list containing all of the pages.<br>
<br>
They are set to either gotten or in-flight.<br>
<br>
In-flight means I have a deferred that's going to go get it (in<br>
get_web_page()).<br>
<br>
IOW, right now, if I don't already have the page, and I have a link<br>
to it, I just start a deferred to go get it.<br>
<br>
Should I limit the number of "in-flight" pages?<br>
<br>
Currently, I'm scanning sites that have upwards of 5000 pages and it<br>
seems that, when I get too many deferred's in flight, the app<br>
*appears* to crash.<br>
<br>
I'm not sure whether it's actually going out to lunch or just appears<br>
that way and, before I go instrumenting the app to death, can anyone<br>
tell me whether there is some sort of practical limit to how many "in-<br>
flight" deferreds might start to cause issues, just due to the sheer<br>
number?<br>
<br>
Thanks for any insight on this that anyone might offer.<br>
<br>
I expect the usual flurry of "you must post your exact code or we<br>
can't help you at all, moron" posts, but...<br>
<br>
In spite of my not having posted specific code, could someone with<br>
some actual experience in this please give me a clue, within an order<br>
of magnitude, how many deferreds might start to cause real trouble?<br>
<br>
Thanks,<br>
<br>
S<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
Twisted-Python mailing list<br>
<a href="mailto:Twisted-Python@twistedmatrix.com">Twisted-Python@twistedmatrix.com</a><br>
<a href="http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python" target="_blank">http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python</a><br>
</blockquote></div><br>