[Twisted-Python] Re: gthreadless.py

Stefano Masini stefano at pragma2000.com
Tue Aug 30 10:36:43 EDT 2005


On 8/30/05, Christopher Armstrong <radeex at gmail.com> wrote:
> The effect that being able to call things that do context switches
> without explicitly marking them as doing so is much farther reaching
> than code readability. I consider it something of a feature that
> deferredGenerator forces you to know when context switches will happen
> at every level, and when I wrote gthreadless I was not intending it to
> be used in a way that didn't require that same knowledge at every
> level. And I don't think that this extra knowledge along the line
> isn't detracting at all to readability, but instead helping it.

I can agree on this.

> And, just for onlookers, I'd like to point out that the code example
> below is not showing the difference between non-explicit-greenlets and
> explicit-defgen, but instead showing the difference between
> explicit-greenlets and plain ond deferreds, which most of us in the

You are perfectly right.
After my first post the discussion went on privately between me and
Nicola, so I should post some of it here, since it contains exactly
such example.

I'll elaborate a bit on my first example, that was as follows (just to
remind and get started with the rest):

@deferredGreenlet
def renderPage(self):
  dataList = [ blockOn(self.backend.callRemote('getDataFromId', elementId))
                      for elementId in self.idList]
  return T.html[ T.body [ 'The result:', T.br, [ (txt, T.br) for txt
in dataList ] ] ]

I basically build a list of values obtained by performing subsequent
calls to callRemote(), everytime passing a parameter from a list.

Let's say that instead of a list of parameters we have a list of
objects, and I build the list of values by calling a method on each of
these objects. Like so:

def renderPage(self):
  dataList = [ element.getData() for element in self.elementList ]
  return T.html[ T.body [ 'The result:', T.br, [ (txt, T.br) for txt
in dataList ] ] ]

As you can see, I took out @deferredGreenlet for now, because it's not needed.
Let's say that self.elementList is made of objects defined like this:

class ElementObject(object):
   def getData(self):
       return 1

Indeed @deferredGreenlet is not needed because I'm not even using pb.

Now, let's say that in a new version of the software I introduce
objects that in order to obtain the result of getData() have to go and
query a remote server through pb. Things get more complicated now
because getData() would return a deferred, while other objects would
return a straight result. This difference is very uncomfortable to
live with because you don't know how to treat the result. The are two
ways out, and in both cases you have to change code you've already
written:
1) return defer.succeed(1) instead of return 1
2) defer.maybeDeferred(element.getData()) instead of element.getData()

In other words, as soon as a blocking method pops up among your
methods, you're forced to change and treat all of them as blocking,
even by making up deferred if needed.
In any case, you're also compelled to change the code of renderPage()
from synchronos style to asynchronous, unless you use waitForDeferred.

I often found myself propagating maybeDeferred's back up several
levels in my code, and I didn't like it, to tell the truth.

On the other hand, if you use greenlets, you can keep renderPage the
same exact way as I wrote it the first time, you just need to decorate
it with @deferredGreenlet. And those methods that perform blocking
calls, simply need to wrap the deferreds with blockOn, and that's it.

Yes, I agree on the following point: code that you though was non
blocking, now can all of a sudden become blocking. In this respect,
yes, a gthreadless implementation that would force you to decorate
every method along the way could help gaining awareness.

But let's think about it: why is it so bad that a method that was
supposed to be non blocking now becomes blocking? I can't think of
anything else than shared resources that now can get accessed
concurrently by other parts of code. Right?
But this problem persists with pure-deferred programming style too!
The problem of concurrent access to shared resources does not depend
on the programming model being synchronous or asynchronous, but simply
on the presence of blocking operations. You have to use locks if you
want to protect a shared resource while you block on a lengthy
operation, no matter what programming model you're using.

So, I hope this example makes my point a little more clear. As I was
saying with Nicola, I don't think gthreadless should be used
everywhere, like it was a solution to some horrible problem with
asynchronous programming. Using deferreds is just great and the
awareness you gain of the internals of your implementation by using
them is just irreplaceable.
But *some* code snippets just come out so much more naturally if you
write them using a synchronous model, that being able to mix the two
is just a terrific feature I think.

cheers,
stefano




More information about the Twisted-Python mailing list