[Twisted-Python] Advice sought on application evolution

Don Dwiggins ddwiggins at advpubtech.com
Fri Mar 21 15:23:08 EDT 2008


glyph at divmod.com wrote in the "Teach Me Twisted Redux" thread:
 > One of the things I did at PyCon was to reach out to the Django
 > community and ask them to start considering what it would take for
 > Twisted to be the preferred container and deployment platform for
 > Django.  They're thinking about it.

I've started thinking about something vaguely similar.  Currently, I 
have a small xml-rpc server in twisted, which provides a few operations 
on a large, complex database for a remote client.  (The bulk of the 
database operations are carried out by a desktop app.)  This was my 
first serious exposure to Twisted, and I found it reasonably easy to get 
going, once I got the hang of the concepts.  (It was probably eased by 
my having previous experience with asynchronous I/O, back in Paleolithic 
times when that was _the_ way it was done.)

My intention is to evolve this server toward a true "middle-tier" layer 
containing the business logic that's now scattered helter-skelter among 
various desktop apps and database stored procedures.  This layer, then, 
will be accessed both by remote clients and "local" (on the same LAN) 
desktop apps.

I'm leaning toward evolving the current xml-rpc server in this
direction, but I'm uncertain whether it's a good idea to implement a 
large amount of business logic directly in an asynchronous paradigm, and 
specifically directly in the Twisted framework.  Twisted looks to be an 
excellent base for a node (loosely speaking) in a communication network, 
but what are the implications of using it for what I have in mind?

A small example of something that bothers me is the necessity to break 
up otherwise "unitary" functions because some parts of them involve 
communications or heavy computation.  More concretely, consider the 
following sketch of a method:

def frobulate(...):
    step_1()
    step_2()
    step_3()
    step_4()
    step_5()
    return result5

Where the steps are logically coherent parts of the frobulation (in 
Beck's terms, the method "smells good").  Now assume that step_3 
involves a database access.  To avoid blocking, I might twist it like this:

def frobulate():
    step_1()
    step_2()
    deferred = step_3_async()
    deferred.addCallback(frobulate_4_and_5)
    return deferred

def frobulate_4_and_5(...):
    step_4()
    step_5()
    return result5

(Where step_3 has been changed to start the access and return a
deferred.)  In this simple sketch, the "visual noise" introduced isn't
too bad, but I'm afraid that as the functions multiply and become more 
complex, it'll make the application considerably harder to understand 
and manage, and possibly more prone to errors and/or unintended 
blocking, where twisting a synchronous operation would require serious 
re-architecting.

Another twist, using generators, might be:

@defer.deferredGenerator
def frobulate():
    step_1()
    step_2()
    step3InProgress = defer.waitForDeferred(step_3_async())
    yield step3InProgress
    step3Rslt = step3InProgress.getResult()
    step_4()
    step_5()
    yield result5
    return

(I've introduced a bit more detail, in that the result of step 3 might
be used in subsequent steps.)

In Python 2.5, I can say this more cleanly:

@defer.inlineCallbacks
def frobulate():
    step_1()
    step_2()
    step3Rslt = yield step_3_async()
    step_4()
    step_5()
    returnValue(result5)

This is looking a lot better, but still the asynchrony of step_3 "shows 
through".  (Again, I'm taking the viewpoint of someone trying to 
understand, maintain, and extend the functionality of the application, 
without having the asynchronous aspects "thrust in his face" more than 
absolutely necessary.)

Of course, I can avoid all this by going to a threading or separate 
process solution, but I'd like to find out if I'm missing something 
before I make such a commitment.

Thanks for any good words,

-- 
Don Dwiggins
Advanced Publishing Technology





More information about the Twisted-Python mailing list