[Twisted-Python] Newbie - how to modify my db-intensive app to useTwisted

Frank Millman frank at chagford.com
Mon Nov 21 02:20:34 EST 2005

> Hi all


> Moving this concept onto Twisted does not seem to work, as it 
> can be running sessions for multiple users, and each 'block' 
> will block the main loop, resulting in a major performance 
> hit. The correct solution, as I understand it, is to change 
> each SQL command to a deferred with a callback. I can do this 
> on the inner layers, where the actual SQL command is 
> executed. But I would also have to do it on each outer layer 
> that calls a function which 'may' trigger a SQL command. 
> There are many of these, and it goes to the heart of my 
> approach of hiding the underlying database complexity from 
> the application layer. Changing this feels like a daunting 
> task, and frankly I would not know where to start.
> The alternative is to use a multi-threaded approach. In this 
> case, I can keep my existing approach largely unchanged, as 
> it does not matter if one thread blocks, the others will 
> continue without being affected.

Thanks for the replies, Marcin and Clark.

I think you have helped to to see a solution, but I would like to clarify

Take the following piece of pseudo code -

    def main():
        result = func1(...)
        do something with result

If func1 blocks, the entire program will be blocked until it returns.

The Twisted approach is -

    def main():
        deferred = func1(...).addCallback(doSomething)

    def doSomething(result):
        do something with result

Now the program will not block, but the key is that main() must return
immediately, thus returning control to Twisted's main loop.

I have many cases where I call a blocking function and wait for a result. I
had envisaged something like this -

Instead of

    def main():
        result1 = func1(...)
        do something with result1
        result2 = func2(...)
        do something with result2

do this

    def main():
        deferred = func1(...).addCallback(doSomething)

    def doSomething(result1):
        do something with result1
        deferred = func2(...).addCallback(doSomethingElse)

    def doSomethingElse(result2):
        do something with result2

If this was the correct approach, it would be a nightmare to go through all
my code and split up each call into a separate deferred and callback.

If I understand Marcin and Clark correctly, I should rather do it this way -

    def main():
        conn = adbapi.ConnectionPool(...)

    def main2():
        result1 = func1(...)
        do something with result1
        result2 = func2(...)
        do something with result2

This involves far fewer changes to my existing code, so I think I could
tackle this. As Marcin says, this is really a multi-threaded solution, so it
is a bit of a cheat, but it looks as if it will work.

Clark, I am using psycopg2, but if I make use of the feature you describe,
would I have to define a separate callback for every SQL command? If so, it
puts me back in my original nightmare scenario. Or have I missed the point?
An example of how I could use this would be very useful.

I hope this makes sense. Any comments will be much appreciated.

Thanks again


More information about the Twisted-Python mailing list