[Twisted-Python] Newbie - how to modify my db-intensive app to useTwisted
Frank Millman
frank at chagford.com
Mon Nov 21 02:20:34 EST 2005
>
> Hi all
>
[snip]
> Moving this concept onto Twisted does not seem to work, as it
> can be running sessions for multiple users, and each 'block'
> will block the main loop, resulting in a major performance
> hit. The correct solution, as I understand it, is to change
> each SQL command to a deferred with a callback. I can do this
> on the inner layers, where the actual SQL command is
> executed. But I would also have to do it on each outer layer
> that calls a function which 'may' trigger a SQL command.
> There are many of these, and it goes to the heart of my
> approach of hiding the underlying database complexity from
> the application layer. Changing this feels like a daunting
> task, and frankly I would not know where to start.
>
> The alternative is to use a multi-threaded approach. In this
> case, I can keep my existing approach largely unchanged, as
> it does not matter if one thread blocks, the others will
> continue without being affected.
>
Thanks for the replies, Marcin and Clark.
I think you have helped to to see a solution, but I would like to clarify
it.
Take the following piece of pseudo code -
def main():
result = func1(...)
do something with result
If func1 blocks, the entire program will be blocked until it returns.
The Twisted approach is -
def main():
deferred = func1(...).addCallback(doSomething)
def doSomething(result):
do something with result
Now the program will not block, but the key is that main() must return
immediately, thus returning control to Twisted's main loop.
I have many cases where I call a blocking function and wait for a result. I
had envisaged something like this -
Instead of
def main():
result1 = func1(...)
do something with result1
result2 = func2(...)
do something with result2
do this
def main():
deferred = func1(...).addCallback(doSomething)
def doSomething(result1):
do something with result1
deferred = func2(...).addCallback(doSomethingElse)
def doSomethingElse(result2):
do something with result2
If this was the correct approach, it would be a nightmare to go through all
my code and split up each call into a separate deferred and callback.
If I understand Marcin and Clark correctly, I should rather do it this way -
def main():
conn = adbapi.ConnectionPool(...)
conn.runInteraction(main2)
def main2():
result1 = func1(...)
do something with result1
result2 = func2(...)
do something with result2
This involves far fewer changes to my existing code, so I think I could
tackle this. As Marcin says, this is really a multi-threaded solution, so it
is a bit of a cheat, but it looks as if it will work.
Clark, I am using psycopg2, but if I make use of the feature you describe,
would I have to define a separate callback for every SQL command? If so, it
puts me back in my original nightmare scenario. Or have I missed the point?
An example of how I could use this would be very useful.
I hope this makes sense. Any comments will be much appreciated.
Thanks again
Frank
More information about the Twisted-Python
mailing list