[Twisted-Python] Lots and lots and lots and lots... of deferreds

[Steve Steiner (listsin)]

On Oct 6, 2009, at 10:57 PM, Glyph Lefkowitz wrote:

> However, you can experiment with it pretty easily using  
> DeferredSemaphore: http://twistedmatrix.com/documents/8.2.0/api/twisted.internet.defer.DeferredSemaphore.html

Cool, I didn't know about that, I'll give it a look.  Thanks!

> If your app is doing something strange that you don't understand,  
> you should instrument it until you understand it.

It's not that I don't understand what's supposed to be happening, or  
that it's doing something strange, it just seems that sites up to  
about 2000 pages work fine, then things get dicey.

I was just looking for some guidance on the "max-deferred" that has  
been found in practical experience more than anything else.

>  Regardless of any practical advice you may receive as a temporary  
> stopgap, there's always a chance that something else is going wrong,  
> and by reducing the number of concurrent requests you're just  
> decreasing its likelihood rather than properly fixing it.

I understand, and agree.  I'm not looking for a stopgap, just maybe a  
ballpark of "don't set more than 2000 in-flight deferreds at one time"  
type of guideline.

I understand that every situation is different, I'm working to limit  
my in-filght requests to a manageable number,

> It's highly unlikely that it's actually the number of Deferreds.  A  
> Deferred is just a Python object, so if you've got the RAM to store  
> them and their associated callbacks, you should be fine.

Yes, I understand that, thank you for clarifying.

> It's more likely that it has something to do with long callback  
> chains, or hitting some kind of file-descriptor limit

The callback chains are short, and I'm not getting a file-descriptor  
limit exception, or any exception that's getting percolated up.

> (what version of Twisted are you using?)

Sorry for not including this earlier...

# python -V
Python 2.6.1

 >>> import twisted
 >>> twisted.__version__
'8.2.0'

I'm running right out of the release versions for these tests since  
that's what my users will have installed.

> or perhaps that 5000 pages is just a lot of pages to request and you  
> might need to wait a while.

Yes, it is a lot of stuff...

What I'm working on determining is whether limiting the number of "in- 
flight" URL getters would beneficial.

Thanks,

S















>
> Good luck,
>
> -Glyph
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20091006/5f6e7844/attachment.htm