[Twisted-Python] Lots and lots and lots and lots... of deferreds
Steve Steiner (listsin)
listsin at integrateddevcorp.com
Wed Oct 7 00:38:55 EDT 2009
On Oct 6, 2009, at 11:25 PM, Andrew Bennetts wrote:
> Steve Steiner (listsin) wrote:
>> I expect the usual flurry of "you must post your exact code or we
>> can't help you at all, moron" posts, but...
> I'll try to restrain myself ;)
Thanks, I appreciate your restraint. Must say, most posts without
code drive me a little cuckoo, too. I hope this one was justified in
not including specific code.
>> My two top guesses are:
> 1) the web server failing to cope with thousands of concurrent
> gracefully, or
Ah, that would certainly make sense in this particular case. I am
asking the server for an *awful* lot of stuff all at one time.
> 2) the number of sockets is hitting a system limit (number of FDs
> you can pass
> to select(), or hitting the max number of file descriptors,
> something like
That may also be an issue, thank you for pointing that out.
> IIRC the RFCs say that clients SHOULD use no more than two
> concurrent connections to a server...
As you said, the performance is going to be rate-limited by the
server's ability to respond to requests anyway, so I think what I'll
do is just make it so that the "get me a page" queue doesn't put more
than a couple of requests "in-flight" at the same time.
Thanks for helping me think this out!
More information about the Twisted-Python