[Twisted-Python] Lots and lots and lots and lots... of deferreds
Steve Steiner (listsin)
listsin at integrateddevcorp.com
Tue Oct 6 23:38:49 EDT 2009
On Oct 6, 2009, at 10:57 PM, Glyph Lefkowitz wrote:
> However, you can experiment with it pretty easily using
> DeferredSemaphore: http://twistedmatrix.com/documents/8.2.0/api/twisted.internet.defer.DeferredSemaphore.html
Cool, I didn't know about that, I'll give it a look. Thanks!
> If your app is doing something strange that you don't understand,
> you should instrument it until you understand it.
It's not that I don't understand what's supposed to be happening, or
that it's doing something strange, it just seems that sites up to
about 2000 pages work fine, then things get dicey.
I was just looking for some guidance on the "max-deferred" that has
been found in practical experience more than anything else.
> Regardless of any practical advice you may receive as a temporary
> stopgap, there's always a chance that something else is going wrong,
> and by reducing the number of concurrent requests you're just
> decreasing its likelihood rather than properly fixing it.
I understand, and agree. I'm not looking for a stopgap, just maybe a
ballpark of "don't set more than 2000 in-flight deferreds at one time"
type of guideline.
I understand that every situation is different, I'm working to limit
my in-filght requests to a manageable number,
> It's highly unlikely that it's actually the number of Deferreds. A
> Deferred is just a Python object, so if you've got the RAM to store
> them and their associated callbacks, you should be fine.
Yes, I understand that, thank you for clarifying.
> It's more likely that it has something to do with long callback
> chains, or hitting some kind of file-descriptor limit
The callback chains are short, and I'm not getting a file-descriptor
limit exception, or any exception that's getting percolated up.
> (what version of Twisted are you using?)
Sorry for not including this earlier...
# python -V
>>> import twisted
I'm running right out of the release versions for these tests since
that's what my users will have installed.
> or perhaps that 5000 pages is just a lot of pages to request and you
> might need to wait a while.
Yes, it is a lot of stuff...
What I'm working on determining is whether limiting the number of "in-
flight" URL getters would beneficial.
> Good luck,
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Twisted-Python