[Twisted-Python] Newbie - how to modify my db-intensive app to use Twisted

glyph at divmod.com glyph at divmod.com
Mon Nov 21 09:31:51 EST 2005

On Sun, 20 Nov 2005 11:59:27 +0200, Frank Millman <frank at chagford.com> wrote:

>Moving this concept onto Twisted does not seem to work, as it can be running
>sessions for multiple users, and each 'block' will block the main loop,
>resulting in a major performance hit. The correct solution, as I understand
>it, is to change each SQL command to a deferred with a callback. I can do
>this on the inner layers, where the actual SQL command is executed. But I
>would also have to do it on each outer layer that calls a function which
>'may' trigger a SQL command. There are many of these, and it goes to the
>heart of my approach of hiding the underlying database complexity from the
>application layer. Changing this feels like a daunting task, and frankly I
>would not know where to start.

The way I would probably work on an application that did this was making each *transaction* into something that ran in a thread and returned a Deferred, which would reduce the footprint of impacted code.  Each user-interface action can be encapsulated as a transaction, and then your code doesn't need to convert each function into a Deferred, just the top-level event handler which kicks off the transaction.

>The alternative is to use a multi-threaded approach. In this case, I can
>keep my existing approach largely unchanged, as it does not matter if one
>thread blocks, the others will continue without being affected.

In the case where you're turning each SQL statement into a Deferred, you're actually using threads anyway; each statement executes in a thread, then sends its result to the main loop.  The important thing when you turn a Twisted application into a multithreaded application is to have a stark distinction between code which runs in threads and code which runs in the main loop.  Code which runs in threads can't call Twisted functions (other than callFromThread), use Deferreds, etc.  It also should refer to as little state as possible, since if you can create isolated groups of objects in the main thread, you don't need to bother with locking as your blocking threaded code runs.

More information about the Twisted-Python mailing list