[Twisted-Python] Re: gthreadless.py

[Stefano Masini]

On 8/30/05, Nicola Larosa <nico at teknico.net> wrote:
> From v.2.0, inside the twisted/internet/defer.py there are one class,
> waitForDeferred, and one function, deferredGenerator, that implement a
> similar pseudo-synchronous style, but using standard generators instead of
> greenlets.

I'm familiar with deferredGenerator. I've been using twisted full time
for two years and a half now and used deferredGenerator for quite a
lot too. That's why I reimplemented gthreadless! :)
 
> Furthermore, PEP 342 has been accepted for v.2.5:
> 
> Coroutines via Enhanced Generators
> http://www.python.org/peps/pep-0342.html
> 
> its enhancements should further simplify such a coding style in Twisted.

The problem with generators and enhanced generators, as I've been
discussing with a few people at Europython, is that they allow you to
jump back and forth between two stack *frames*. Whereas greenlets
support jumping among *full* stacks. This means that from within a
@deferredGreenlet'ed function you can make calls to other functions
that call blockOn, while from a @deferredGenerator'ed function you
can't call a method that in turn calls waitForDeferred.
 
While this may seems quite a subtle difference, I think it is not if
you look at it from the perspective of code readability -- let alone
the 3 liner horrible hack that waitForDeferred forces you to, at least
until PEP 342, as we all know.
 
> However, a couple of recent blog entries show that this way of "hiding"
> Deferreds raises some eyebrows within Twisted's inner circle:
> 
> Magical Concurrency Faeries or How I Learned To Stop Worrying and Love
> Deferreds
> http://www.livejournal.com/users/jcalderone/9531.html
> 
> Knowing Santa Claus is Fake Doesn't Ruin Christmas
> http://www.livejournal.com/users/glyf/40037.html
> 
> Personally, I think that while explicitly specifying deferreds and
> callbacks and errbacks can be quite verbose, and may sometimes obscure the
> program flow, the comfort of seeing clearly the boundaries of each
> uninterruptible execution unit makes it worthwhile.
> 

Believe me. I'm not one of those users in the "periphery of the
Twisted community" thinking that asynchronous programming is too hard.
I've been writing network code for quite a while and I matured the
idea that threads get in your way back when I still didn't know
python.
What I'm only concerned about now is coding *style*. Making code
better looking, thus more easily maintainable.
I rewrote gthreadless because having a usable implementation allowed
rewriting parts of my existing (big) application in a much simpler
way. And let me stress this: *parts* of it. I think gthreadless should
only be used here and there, not everywhere. One should always keep
thinking of deferreds, and even inside a @deferredGreenlet'ed function
one should be very clear that blockOn() really only spits back a
deferred to the reactor. But at least, you can debug your function
without having to jump back and forth 40 lines at a time just to get
to the proper callback or errback. I think sometimes this verbosity
may get in the way of the pythonic spirit of keeping stuff simple.
 
Here is an example that makes justice to this approach. It involves
Perspective Broker. Think of writing a web frontend to an application
on the backend that exports functionality through pb. (the code may
not work, I'm just making it up now without testing it)
(I hope everybody is familiar with nevow.stan. If not, take a look at
it. It's worth.)
 
@deferredGreenlet
def renderPage(self):
    dataList = [ blockOn(self.backend.callRemote('getDataFromId', elementId))
                        for elementId in self.idList]
    return T.html[ T.body [ 'The result:', T.br, [ (txt, T.br) for txt
in dataList ] ] ]

Without gthreadless:

def renderPage(self):
    dataList = []
    def fetchDataRemotely(elementList):
        def cbFetch(elementData):
            dataList.append(elementData)
            if len(elementList) > 0:
                return fetchDataRemotely(elementList[1:])
        return self.backend.callRemote('getDataFromId',
elementId).addCallback(cbFetch)
    def cb(crap):
        return T.html[ T.body [ 'The result:', T.br, [ (txt, T.br) for
txt in dataList ] ] ]
    return fetchDataRemotely(self.idList).addCallback(cb)

I hope everyone agrees that the level of complexity in *reading* and
*understanting* what the above code snippets do is not the same.
The above code could have actually been written the same way using
deferredGenerator and PEP 342, or in a slightly more verbose way
without PEP 342. But the example is simple. If instead of simply
callRemote() you had to use another method, that in turn needed
callRemote, maybe a couple of times (very possible if you use pb),
then greenlets would have been indispensable, in order to keep the
renderPage() the same as you saw.

Cheers,
stefano