[Twisted-Python] Advice sought on application evolution

[washort at twistedmatrix.com]

On Fri, Mar 21, 2008 at 12:23:08PM -0700, Don Dwiggins wrote:
> A small example of something that bothers me is the necessity to break 
> up otherwise "unitary" functions because some parts of them involve 
> communications or heavy computation.  More concretely, consider the 
> following sketch of a method:
> def frobulate(...):
>    step_1()
>    step_2()
>    step_3()
>    step_4()
>    step_5()
>    return result5
> 
> Where the steps are logically coherent parts of the frobulation (in 
> Beck's terms, the method "smells good").  Now assume that step_3 
> involves a database access.  To avoid blocking, I might twist it like this:
> 
> def frobulate():
>    step_1()
>    step_2()
>    deferred = step_3_async()
>    deferred.addCallback(frobulate_4_and_5)
>    return deferred
> 
> def frobulate_4_and_5(...):
>    step_4()
>    step_5()
>    return result5
> 
> (Where step_3 has been changed to start the access and return a
> deferred.)  In this simple sketch, the "visual noise" introduced isn't
> too bad, but I'm afraid that as the functions multiply and become more 
> complex, it'll make the application considerably harder to understand 
> and manage, and possibly more prone to errors and/or unintended 
> blocking, where twisting a synchronous operation would require serious 
> re-architecting.

There's a significant advantage to this approach you may not have
considered: When reading frobulate(), you can be assured that no other
code is being executed between step_1(), step_2(), and step_3_async().
Between step_3_async and step_4, anything may happen and arbitrary
amounts of time can pass. Having this explicitly represented in the code
makes certain things /much/ easier to reason about. I agree that the
syntax can be a little unwieldy, but the basic idea is sound: if you are
going to stop executing a multi-step operation and continue it later, it
should be clear that the flow of execution doesn't directly pass between
the two parts of the operation. There are other languages that make this
easier to express, but Python has enough advantages that I find this
problem worth putting up with. :)

><snip code example with generators>
> This is looking a lot better, but still the asynchrony of step_3 "shows 
> through".  (Again, I'm taking the viewpoint of someone trying to 
> understand, maintain, and extend the functionality of the application, 
> without having the asynchronous aspects "thrust in his face" more than 
> absolutely necessary.)

It does show through, and it has to, to make the program comprehensible.
I have a mild dislike for inlineCallbacks because it _does_ minimize the
changes between 'sequential' and 'twisted' code. :)

Basically, there are two choices in this regard for concurrency:

1) Preserve the property that only code you can see locally referenced
   can be executed (i.e., that in "step1(); step2()" only code called
   by those two functions can be executed before the statement is
   finished) and complicate the syntax in places where other code needs
   to be allowed to execute.

2) Preserve the syntax of statements being executed in order but give up
   the guarantee that nothing will be changed by 'outside' code in
   between them.

Twisted's design is centered entirely around #1. Multithreaded
concurrency chooses #2, which is why there's much need for locks and
mutexes and monitors and what have you. The result is slightly more
verbose code, but much more _understandable_ code.
 
> Of course, I can avoid all this by going to a threading or separate 
> process solution, but I'd like to find out if I'm missing something 
> before I make such a commitment.

Separate processes can be the best of both worlds -- things can run in
sequence and not affect each other in surprising ways. The downside is
that it's more difficult for them to affect each other at all. :)

I hope this helps explain some of Twisted's design choices.