[Twisted-Python] A kinder and more consistent defer.inlineCallbacks

Terry Jones terry at jon.es
Fri Nov 21 13:04:42 EST 2008

Here's a suggestion for making inlineCallbacks more consistent and less
confusing.  Let's suppose you're writing something like this:

    def func():

    result = func()

There are 2 things that could be better, IMO:

1. func may not yield. In that case, you either get an AttributeError when
inlineCallbacks tries to send(). Or worse, the call to send might actually
work, and do who knows what. I.e., func() could return an object with a
send method but which is not a generator. For some fun, run some code that
calls the following decorated function (see if you can figure out what will
happen before you do):

    def f():
        class yes():
            def send(x, y):
                print 'yes'
                # accidentally_destroy_the_universe_too()
        return yes()

2. func might raise before it get to its first yield. In that case you'll
get an exception thrown when the inlineCallbacks decorator tries to create
the wrapper function:

    File "/usr/lib/python2.5/site-packages/twisted/internet/defer.py", line 813, in unwindGenerator
      return _inlineCallbacks(None, f(*args, **kwargs), Deferred())

There's a simple and consistent way to handle both of these. Just have
inlineCallbacks do some initial work based on what it has been passed:

    def altInlineCallbacks(f):
        def unwindGenerator(*args, **kwargs):
            deferred = defer.Deferred()
                result = f(*args, **kwargs)
            except Exception, e:
                return deferred
            if isinstance(result, types.GeneratorType):
                return defer._inlineCallbacks(None, result, deferred)
            return deferred

        return mergeFunctionMetadata(f, unwindGenerator)

This has the advantage that (barring e.g., a KeyboardInterrupt in the
middle of things) you'll *always* get a deferred back when you call an
inlineCallbacks decorated function. That deferred might have already called
or erred back (corresponding to cases 1 and 2 above).

I'm going to use this version of inlineCallbacks in my code.  There's a
case for it making it into Twisted itself: inlinecallbacks is already
cryptic enough in its operation that anything we can do to make its
operation more uniform and less surprising, the better.

You might think that case 1 rarely comes up. But I've hit it a few times,
usually when commenting out sections of code for testing. If you
accidentally comment out the last yield in func, it no longer returns a
generator and that causes a different error.

And case 2 happens to me too. Having inlinecallbacks try/except the call to
func is nicer because it means I don't have to be quite so defensive in
coding. So instead of me having to write

        d = func()
    except Exception:
        # ???

and try to figure out what happened if an exception fired, I can just write
d = func() and add errbacks as I please (they then have to figure out what
happened).  The (slight?) disadvantage to my suggestion is that with the
above try/except fragment you can tell if the call to func() raised before
ever yielding. You can detect that, if you need to, with my approach if
you're not offended by looking at d.called immediately after calling func.

The alternate approach also helps if you're a novice, or simply being
lazy/careless/forgetful, and writing:

    d = func()

thinking you have your ass covered, but you actually don't (due to case 2).

There's some test code attached that illustrates all this.


-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: altInlineCallbacks.py
Url: http://twistedmatrix.com/pipermail/twisted-python/attachments/20081121/ab91d7f8/attachment.txt 

More information about the Twisted-Python mailing list