[Twisted-Python] Scaling problem
glyph at twistedmatrix.com
Tue Nov 4 15:20:46 EST 2003
Uwe C. Schroeder wrote:
> Is there a way to have twisted do something like apache or postgres and split
> the incoming connection to several processes ? It's not a web application,
> I'm solely using pb.
This is possible, although not as easy as I'd like it to be. In
principle, on most UNIX-based OSes, it's possible to listen() and
accept() in one process, pass the returned file descriptor to a
subprocess and do communication there. I'd like to make this automatic
at some point, but won't have the time to do it for at least another few
The fact that it's not as graceful or automatic as I like doesn't mean
it's impossible though. If you want to just have one "control" process
that shuttles around I/O and N worker processes that do the actual work,
you can still use spawnProcess and communicate over stdin/stdout to the
subprocesses about new connections using a simple protocol which
multiplexes multiple socket connections down to the one pipe connection
Also, if database latency is your main problem, or you can move a little
bit of performance-sensitive code into C and relinquish the GIL while
they're happening, you can just use callInThread to run your
transactions in threads instead of having the reactor waiting on them.
Is this helpful?
More information about the Twisted-Python