[Twisted-Python] Advice sought on application evolution
Don Dwiggins
ddwiggins at advpubtech.com
Fri Mar 21 15:23:08 EDT 2008
glyph at divmod.com wrote in the "Teach Me Twisted Redux" thread:
> One of the things I did at PyCon was to reach out to the Django
> community and ask them to start considering what it would take for
> Twisted to be the preferred container and deployment platform for
> Django. They're thinking about it.
I've started thinking about something vaguely similar. Currently, I
have a small xml-rpc server in twisted, which provides a few operations
on a large, complex database for a remote client. (The bulk of the
database operations are carried out by a desktop app.) This was my
first serious exposure to Twisted, and I found it reasonably easy to get
going, once I got the hang of the concepts. (It was probably eased by
my having previous experience with asynchronous I/O, back in Paleolithic
times when that was _the_ way it was done.)
My intention is to evolve this server toward a true "middle-tier" layer
containing the business logic that's now scattered helter-skelter among
various desktop apps and database stored procedures. This layer, then,
will be accessed both by remote clients and "local" (on the same LAN)
desktop apps.
I'm leaning toward evolving the current xml-rpc server in this
direction, but I'm uncertain whether it's a good idea to implement a
large amount of business logic directly in an asynchronous paradigm, and
specifically directly in the Twisted framework. Twisted looks to be an
excellent base for a node (loosely speaking) in a communication network,
but what are the implications of using it for what I have in mind?
A small example of something that bothers me is the necessity to break
up otherwise "unitary" functions because some parts of them involve
communications or heavy computation. More concretely, consider the
following sketch of a method:
def frobulate(...):
step_1()
step_2()
step_3()
step_4()
step_5()
return result5
Where the steps are logically coherent parts of the frobulation (in
Beck's terms, the method "smells good"). Now assume that step_3
involves a database access. To avoid blocking, I might twist it like this:
def frobulate():
step_1()
step_2()
deferred = step_3_async()
deferred.addCallback(frobulate_4_and_5)
return deferred
def frobulate_4_and_5(...):
step_4()
step_5()
return result5
(Where step_3 has been changed to start the access and return a
deferred.) In this simple sketch, the "visual noise" introduced isn't
too bad, but I'm afraid that as the functions multiply and become more
complex, it'll make the application considerably harder to understand
and manage, and possibly more prone to errors and/or unintended
blocking, where twisting a synchronous operation would require serious
re-architecting.
Another twist, using generators, might be:
@defer.deferredGenerator
def frobulate():
step_1()
step_2()
step3InProgress = defer.waitForDeferred(step_3_async())
yield step3InProgress
step3Rslt = step3InProgress.getResult()
step_4()
step_5()
yield result5
return
(I've introduced a bit more detail, in that the result of step 3 might
be used in subsequent steps.)
In Python 2.5, I can say this more cleanly:
@defer.inlineCallbacks
def frobulate():
step_1()
step_2()
step3Rslt = yield step_3_async()
step_4()
step_5()
returnValue(result5)
This is looking a lot better, but still the asynchrony of step_3 "shows
through". (Again, I'm taking the viewpoint of someone trying to
understand, maintain, and extend the functionality of the application,
without having the asynchronous aspects "thrust in his face" more than
absolutely necessary.)
Of course, I can avoid all this by going to a threading or separate
process solution, but I'd like to find out if I'm missing something
before I make such a commitment.
Thanks for any good words,
--
Don Dwiggins
Advanced Publishing Technology
More information about the Twisted-Python
mailing list