[Twisted-Python] Twisted's ability to handle many open connections.
tcj25 at cam.ac.uk
Wed Apr 26 19:30:00 EDT 2006
>>>>> "glyph" == glyph <glyph at divmod.com> writes:
glyph> That's more or less accurate. It sounds like you understand this,
glyph> but because I can never repeat it enough: Deferreds don't do
glyph> anything magic or get involved with select() directly
Ok, got it.
glyph> On Windows, there is a reactor that supports a maximum of 63
glyph> concurrent events (assuming your process has no windows open). On
glyph> UNIX the default reactor is still select() based and will definitely
glyph> bog down if you have too many simultaneous sockets open. Sometimes
glyph> it is handled quite badly!
glyph> However, the selection of a reactor is a configuration option, not
glyph> an application-architecture issue (unless, of course, you are
glyph> writing a GUI which has requirements for what event loop it will run
glyph> You can choose a reactor with the -r argument to twistd, trial, and
glyph> other similar tools like axiomatic. You can choose different
glyph> reactors for different platforms. Application code should be none
glyph> the wiser.
Thanks for the details. Sounds like I can handle the problem, if it ever
happens, by simply bumping FD_SETSIZE and rebuilding the kernel, then
proceeding as normal. And/or drop old connections.
glyph> If your application has such a case, though, implementing an epoll,
glyph> aio, kqueue, libevent, or <insert your totally non-portable,
glyph> OS-specific, crazily scalable IO multiplexing API here> reactor
glyph> should not be an inordinate amount of work. Less work, at least,
glyph> than building such a beast and then writing what amounts to 2/3 of
glyph> the Twisted core around it anyway :).
Ok, great. I'm actually backing my way out of using lots of already written
low-level asynchronous networking code, my own RPC mechanism, etc., in
favor of letting someone else do that work :-) So I do know what you mean,
More information about the Twisted-Python