[Twisted-Python] TCP Proxy scalability issue

[Mohapatra Saurav]

Hi,

 

We're currently running a TCP proxy  / port multiplexer written in
Twisted to route to multiple worker server processes running on loopback
ports.

 

The architecture is something like this

 

Worker processes W1...n run listening on P1..n on the loopback and one
router process (twisted based) runs on public port P0 exposed to the
real world.

 

The clients connect to P0 and the first few bytes they send indicates
which worker process they wish to connect to. The Twisted protocol
implementation then creates a relay TCP connection to loopback worker
process port and after that forwards received data on P0 to the worker
port on loop back and sends back data received from worker port to the
external connection.

 

We're noticing significant degradation / starvation of the clients under
load (around 25 concurrent connections are enough to simulate this).

 

We're running on Linux (CentOS 5.2) using python 2.5 and Twisted latest
source tarball using the epoll reactor and all settings are default.
Each connection sends back around 8-24kb data per second.

 

My question is are there any "production" settings I need to do
(threadpool etc.) to get the maximum out of twisted.

 

Regards

Saurav

 

 

 

Saurav Mohapatra
Director of Technology


(  508.875.9501  | Mobile: 774.242.7422
*  saurav at dimdim.com
Skype: mohaps   |   AIM: mohaps

    Meet freely.

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20090501/d4a6327e/attachment.htm 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 6358 bytes
Desc: image001.png
Url : http://twistedmatrix.com/pipermail/twisted-python/attachments/20090501/d4a6327e/attachment.png