<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On Oct 6, 2009, at 10:57 PM, Glyph Lefkowitz wrote:</div><div><br></div><blockquote type="cite"><div class="gmail_quote"><div>However, you can experiment with it pretty easily using DeferredSemaphore: <a href="http://twistedmatrix.com/documents/8.2.0/api/twisted.internet.defer.DeferredSemaphore.html">http://twistedmatrix.com/documents/8.2.0/api/twisted.internet.defer.DeferredSemaphore.html</a><br></div></div></blockquote><div><br></div><div>Cool, I didn't know about that, I'll give it a look. Thanks!</div><div><br></div><blockquote type="cite"><div class="gmail_quote"><div>If your app is doing something strange that you don't understand, you should instrument it until you understand it.</div></div></blockquote><div><br></div><div>It's not that I don't understand what's supposed to be happening, or that it's doing something strange, it just seems that sites up to about 2000 pages work fine, then things get dicey.</div><div><br></div><div>I was just looking for some guidance on the "max-deferred" that has been found in practical experience more than anything else.</div><div><br></div><blockquote type="cite"><div class="gmail_quote"><div> Regardless of any practical advice you may receive as a temporary stopgap, there's always a chance that something <i>else</i> is going wrong, and by reducing the number of concurrent requests you're just decreasing its likelihood rather than properly fixing it.<br></div></div></blockquote><div><br></div><div>I understand, and agree. I'm not looking for a stopgap, just maybe a ballpark of "don't set more than 2000 in-flight deferreds at one time" type of guideline.</div><div><br></div><div>I understand that every situation is different, I'm working to limit my in-filght requests to a manageable number, </div><div><br></div><blockquote type="cite"><div class="gmail_quote"><div>It's highly unlikely that it's actually the number of Deferreds. A Deferred is just a Python object, so if you've got the RAM to store them and their associated callbacks, you should be fine. </div></div></blockquote><div><br></div><div>Yes, I understand that, thank you for clarifying.</div><div><br></div><blockquote type="cite"><div class="gmail_quote"><div>It's more likely that it has something to do with long callback chains, or hitting some kind of file-descriptor limit </div></div></blockquote><div><br></div><div>The callback chains are short, and I'm not getting a file-descriptor limit exception, or any exception that's getting percolated up.</div><div><br></div><blockquote type="cite"><div class="gmail_quote"><div>(what version of Twisted are you using?) </div></div></blockquote><div><br></div><div><div>Sorry for not including this earlier...</div><div><br></div><div># python -V</div><div>Python 2.6.1</div><div><br></div><div><div>>>> import twisted</div><div>>>> twisted.__version__</div><div>'8.2.0'</div><div><br></div></div></div><div>I'm running right out of the release versions for these tests since that's what my users will have installed.</div><div><br></div><blockquote type="cite"><div class="gmail_quote"><div>or perhaps that 5000 pages is just a lot of pages to request and you might need to wait a while.<br></div></div></blockquote><div><br></div><div><div>Yes, it is a lot of stuff...</div><div><br></div><div>What I'm working on determining is whether limiting the number of "in-flight" URL getters would beneficial.</div><div><br></div><div>Thanks,</div><div><br></div><div>S</div><div><br></div><div><br></div><div><br></div><div><br></div></div><div><br></div><div><br></div><div><div><br></div></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><br><blockquote type="cite"><div class="gmail_quote"><div>
<br>Good luck,<br><br>-Glyph<br></div></div>
_______________________________________________<br>Twisted-Python mailing list<br><a href="mailto:Twisted-Python@twistedmatrix.com">Twisted-Python@twistedmatrix.com</a><br>http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python<br></blockquote></div><br></body></html>