[Twisted-Python] Lots and lots and lots and lots... of deferreds

[Steve Steiner (listsin)]

On Oct 6, 2009, at 11:25 PM, Andrew Bennetts wrote:

> Steve Steiner (listsin) wrote:
> [...]
>> 	I expect the usual flurry of  "you must post your exact code or we
>> can't help you at all, moron" posts, but...
>
> I'll try to restrain myself ;)

Thanks, I appreciate your restraint.  Must say, most posts without  
code drive me a little cuckoo, too.  I hope this one was justified in  
not including specific code.

>> My two top guesses are:
>
> 1) the web server failing to cope with thousands of concurrent  
> requests
>    gracefully, or

Ah, that would certainly make sense in this particular case.  I am  
asking the server for an *awful* lot of stuff all at one time.

> 2) the number of sockets is hitting a system limit (number of FDs  
> you can pass
>    to select(), or hitting the max number of file descriptors,  
> something like
>    that)

That may also be an issue, thank you for pointing that out.

> IIRC the RFCs say that clients SHOULD use no more than two  
> concurrent connections to a server...

As you said, the performance is going to be rate-limited by the  
server's ability to respond to requests anyway, so I think what I'll  
do is just make it so that the "get me a page" queue doesn't put more  
than a couple of requests "in-flight" at the same time.

Thanks for helping me think this out!

Thanks,

S