[Twisted-Python] scaling with lots of clients

[Grant Mckenzie]

Hi,

I have a twisted Perspective Broker based server. Recently with an increased number of clients we are seeing a decrease in throughput. Profiling the server it seems that it is spending the majority of time sending messages to the clients (unsuprisingly).

Messages are sent to the client from the server using a callRemote, invoked from the  event queue using reactor.callFromThread - nice and simple. What designs have people used to improve throughput in a heavily loaded server? My experience is telling me that I need to implement some sort of thread/process pool to divide the messaging to clients. Various documents on the twisted website strongly suggets that this is not a good idea. Thoughts? 

Cheers,
G.