[Twisted-Python] freeing the reactor to do other jobs

[Reza Lotun]

On Fri, Nov 7, 2008 at 1:29 PM, Jeff Dyke <jeff.dyke at gmail.com> wrote:
> I want the caller to wait on a result from this process, but I also
> want the reactor to be able to handle other requests as they come in.
> This function is not directly registered in the xmlrpc server with
> xmlrpc_getData, but is called by that type of method after validation
> that it is allowed to run in this context.

Hi Jeff,

I'll let others tackle your specific twisted-database integration
problem, but here are a few general rules of them I've found useful
when working with twisted, sqlite and wx.

- Any long-computations will block your app in general because of the
GIL. Threading won't solve this (unless you move the compute intensive
code into a C-module which explicitly releases the GIL, which can be
hairy). Investigate using reactor.spawnProcess to spawn external
processes to do any compute intensive stuff.

- Try using a database library which releases the GIL - I've found
great gains by using apsw instead of the built-in sqlite3 library

- To integrate I/O blocking operations, it's useful to use
deferToThread - which will block on the operation in another thread in
Twisted's threadpool. Just make sure if you're in other threads you
*schedule* calls to the main twisted thread via reactor.callFromThread
(and similarly in you're running wx and twisted in separate threads
via wx.CallAfter)

I hope this is useful.

Cheers,
Reza


-- 
Reza Lotun
Senior Software Engineer
GetPeer Limited
reza at getpeer.com