[Twisted-Python] epoll and other questions

Andrea Arcangeli andrea at cpushare.com
Wed Oct 6 15:24:54 EDT 2004

Is there any plan to use epoll instead of poll to make twisted scalabile
with hundred thousand simultanous sockets connected?

I'm evaluating if to use twisted or C/C++ for my cpushare.com server
side. I'd expect network to be the main bottleneck in the short term,
and so I feel safe at spending extra cycles in userspace (at least in
the short term), but even in the short it should at least handle hundred
thousand tcp connections, and so I'm really scared about poll. I think
poll would be the biggest showstopper, and that's why I'm asking to make
twisted use epoll. I assume my application would require no change, so I
can start developing with current twisted, I can test it with poll, and
then later fix the internals when the slowdown becomes noticeable.  Right?

I understand there's no limitation on the number of sockets
simultanously open, I just need to use ulimit to boost the limit of fds.

A slightly separated issue: I assume it's best for me not to do any
blocking I/O in the main network server handling the 100k connections and
to create a secondary internal server communicating again through tcp/ip
(loopback device) with the primary server to do the real blocking I/O.
Is this correct? Best would be to use asynchronous I/O for the IO, but I
think using a second process will be a lot simpler in practice since I
don't need bulk I/O performance (I only need to avoid blocking). I only
want to keep the network pipeline full even when some disk-read is
happening. Best would be to use threading (or shared memory with
MAP_SHARED in tmpfs), but it seems twisted is not mature enough for
threading and shared memory communication using futex, right?  If I
would write it in C I could probably get various performance bits
faster but I doubt the time spent on those bits would payoff
significantly, opinions? 

Another thing I plan doing is to ship the public key (matching the
private key stored only on the server) on the client source tarball,
this way as far as people downloaded the right tarball, they will be
able to securely connect to the server since they will be able to check
the signature. Is there any example of this idea (public key stored in a
file in the client package) available somewhere?

thank you!

PS. if somebody has some spare time and wants to have a look at the
python _client_ code downloadable at www.cpushare.com/download.php and
tell me if it's decent code and if I interfaced correctly between
twisted and pyqt that's welcome. I'm at the point that I've to open up
the connections from the QT event handlers etc.. I still have some
unsolved issue on the QT side (like error windows instead of using
sys.stderr and disabling one side of the tab when the other is running),
but those are low priority at the moment, the GUI design was
strightforward and the real big issues will start now in the network
protocol implementation with twisted.

More information about the Twisted-Python mailing list