[Twisted-Python] Advise for heavy concurrency

[Atilla]

I'm interested in how well Twisted would perform in situations where
there is a very high number of concurrent connections.

The problem: I'm working on a feature that would require me to
distribute small amounts, both in occurrence and size, of packets to a
very large number of connected clients. Think in the terms of a Jabber
or IRC server, but with a lot less traffic. What we're trying to do is
to maximize the number of simultaneous connections a server can run,
to an absolute maximum. It'll be a dedicated system so crippling any
system-wide setting is not going to interfere with other applications.

So far experimenting with the epoll reactor has shown really nice
scalability as numbers grow. Thousands of connections per machine are
no problem at all, but we need to scale out heavily.

At a certain point the memory requirements for keeping open all those
connections in the system grow higher, but those can be minimized by
adjusting system buffer sizes, tcp window sizes and certain other
settings.

Furthermore - more than one instance of the service could be ran per
server, dedicating two or three processes on a 4-core CPU system,
until exhausting the rest of the limits, in case CPU oevrhead becomes
large.

I'm also considering advantages that might come through different I/O
and/or process schedulers. Is there anything that would be beneficial
to my situation, considering the server will be a dedicated machine
with little to nothing else to run on it that might suffer performance
degradation?

Does anyone have any experience with scaling twisted to a very large
number of parallel connections? Are there any limitations that I might
be missing?

We'll start experimenting with a sample service soon to try to
identify any specific limitations that will need to be worked on.

Any tips are wellcome. Cheers,