[Twisted-Python] freeing the reactor to do other jobs
exarkun at divmod.com
Fri Nov 7 09:07:10 EST 2008
On Fri, 7 Nov 2008 13:56:53 +0000, Reza Lotun <reza at getpeer.com> wrote:
>On Fri, Nov 7, 2008 at 1:29 PM, Jeff Dyke <jeff.dyke at gmail.com> wrote:
>> I want the caller to wait on a result from this process, but I also
>> want the reactor to be able to handle other requests as they come in.
>> This function is not directly registered in the xmlrpc server with
>> xmlrpc_getData, but is called by that type of method after validation
>> that it is allowed to run in this context.
>I'll let others tackle your specific twisted-database integration
>problem, but here are a few general rules of them I've found useful
>when working with twisted, sqlite and wx.
>- Any long-computations will block your app in general because of the
>GIL. Threading won't solve this (unless you move the compute intensive
>code into a C-module which explicitly releases the GIL, which can be
>hairy). Investigate using reactor.spawnProcess to spawn external
>processes to do any compute intensive stuff.
Long-running computations generally only block the thread they're
running in. This shouldn't be surprising, since there's not really
any difference between a function that does a "computation" and a
function which is somehow just "regular" Python code and isn't a
"computation". You can find details of how threading works in
Python in the Python documentation, but briefly, after N bytecodes
are executed, the VM running in thread A releases the GIL (required
to execute bytecode) and any other thread has a chance to acquire it.
By default, N is 100 (enough to multiply about 30 numbers together).
It's only when you bring C into the picture that you have to think
about explicitly releasing the GIL to prevent one thread from blocking
all the rest. As long as your program is all Python, all your threads
will basically play nicely together (an exception to this seems to be
time.sleep() on Windows sometimes, which many people find blocks all
threads inexplicably - but time.time is implemented in C, so this really
just proves the point :).
More information about the Twisted-Python