[Twisted-Python] twisted application server

glyph at divmod.com glyph at divmod.com
Wed Feb 27 19:20:29 EST 2008

On 11:37 pm, clemesha at gmail.com wrote:
>Would you say that this method of using a separate process instead of
>threads to do work also is possibly good for database operations?

>I'm always reading 'beware of threads' ;) with respect to Twisted,
>so using a very simple control protocol to manage a separate
>dedicate database process might be better than
>relying on the threading that 'adbapi' uses?

It would definitely work.  The main reason adbapi itself isn't done this 
way is that setting up a process pool is unfortunately more work than 
spawning a thread - you have to decide on a proper set of environment 
variables, locate the python interpreter, locate the script that you're 
going to run, make sure that script is installed by setup.py, decide on 
a control protocol, wait for the subprocess to start up before sending 
it messages, shut down the process appropriately, catch process 
termination and restart (or not), etc etc etc.  Twisted should really 
have a nice API that does all that stuff for you, though, and it's a 
shame that it doesn't.

The other reason is that adbapi is old.  If we were going to implement 
something like ADBAPI today, we'd probably write a process pool first, 
but adbapi was written as a quick hack to get some database integration 
a long time ago.

The only caveat is that additional Python interpreters take up more RAM 
than additional threads.  If the database processes are doing any heavy 
CPU lifting though, this cost could well be worth it.

More information about the Twisted-Python mailing list