[Twisted-Python] A kinder and more consistent defer.inlineCallbacks

[Drew Smathers]

On Fri, Nov 21, 2008 at 1:04 PM, Terry Jones <terry at jon.es> wrote:
> Here's a suggestion for making inlineCallbacks more consistent and less
> confusing.  Let's suppose you're writing something like this:
>
>    @inlineCallbacks
>    def func():
>        ....
>
>    result = func()
>
> There are 2 things that could be better, IMO:
>
> 1. func may not yield. In that case, you either get an AttributeError when
> inlineCallbacks tries to send(). Or worse, the call to send might actually
> work, and do who knows what. I.e., func() could return an object with a
> send method but which is not a generator. For some fun, run some code that
> calls the following decorated function (see if you can figure out what will
> happen before you do):
>
>    @defer.inlineCallbacks
>    def f():
>        class yes():
>            def send(x, y):
>                print 'yes'
>                # accidentally_destroy_the_universe_too()
>        return yes()
>

Why not just return a Deferred from the function and not decorate it.
Or document the function as returning a value if it doesn't block.

> 2. func might raise before it get to its first yield. In that case you'll
> get an exception thrown when the inlineCallbacks decorator tries to create
> the wrapper function:
>
>    File "/usr/lib/python2.5/site-packages/twisted/internet/defer.py", line 813, in unwindGenerator
>      return _inlineCallbacks(None, f(*args, **kwargs), Deferred())
>
>
> There's a simple and consistent way to handle both of these. Just have
> inlineCallbacks do some initial work based on what it has been passed:
>
>    def altInlineCallbacks(f):
>        def unwindGenerator(*args, **kwargs):
>            deferred = defer.Deferred()
>            try:
>                result = f(*args, **kwargs)
>            except Exception, e:
>                deferred.errback(e)
>                return deferred
>            if isinstance(result, types.GeneratorType):
>                return defer._inlineCallbacks(None, result, deferred)
>            deferred.callback(result)
>            return deferred
>
>        return mergeFunctionMetadata(f, unwindGenerator)

Essentially this is equivalent to _not_ decorating your function and
returning a Deferred via defer.succeed(value) or defer.fail(error).
I'm still not sure why it's necessary to stitch this kind of behavior
into inlineCallbacks().  From my perspective inlineCallbacks() simply
equates: "I want to do more than one asynchronous operation inline and
here's some syntactic sugar using yield as an expression."

>
> This has the advantage that (barring e.g., a KeyboardInterrupt in the
> middle of things) you'll *always* get a deferred back when you call an
> inlineCallbacks decorated function. That deferred might have already called
> or erred back (corresponding to cases 1 and 2 above).
>

You will always get a Deferred  back if the function is a generator.

> I'm going to use this version of inlineCallbacks in my code.  There's a
> case for it making it into Twisted itself: inlinecallbacks is already
> cryptic enough in its operation that anything we can do to make its
> operation more uniform and less surprising, the better.
>
> You might think that case 1 rarely comes up. But I've hit it a few times,
> usually when commenting out sections of code for testing. If you
> accidentally comment out the last yield in func, it no longer returns a
> generator and that causes a different error.

That's what unit tests are for :)  But I'm not sure how you could
easily wind up in this scenario considering generators don't allow
return with an argument.

Drew