[Twisted-Python] A kinder and more consistent defer.inlineCallbacks

Terry Jones terry at jon.es
Fri Nov 21 16:17:23 EST 2008

>>>>> "Drew" == Drew Smathers <drew.smathers at gmail.com> writes:
Drew> On Fri, Nov 21, 2008 at 1:04 PM, Terry Jones <terry at jon.es> wrote:

Drew> Why not just return a Deferred from the function and not decorate it.
Drew> Or document the function as returning a value if it doesn't block.

In that case (1) I'm considering what happens if you take the last yield
out of a function decorated with inlineCallbacks. You might do it via
commenting something out for testing, or you might do it just as a matter
of course in changing the behavior of the function.

If you do either of those, inlineCallbacks will raise an exception. With
the suggestion I sent, it will just run as expected: the original function
will be called, and you'll get the value it returns back as the result of
the returned deferred.

Drew> Essentially this is equivalent to _not_ decorating your function and
Drew> returning a Deferred via defer.succeed(value) or defer.fail(error).
Drew> I'm still not sure why it's necessary to stitch this kind of behavior
Drew> into inlineCallbacks().

It's not necessary, it just makes inlineCallbacks gracefully cover two
cases that it currently throws exceptions on, and it makes inlineCallbacks
always return a deferred, which is arguably also an advantage.

Drew> From my perspective inlineCallbacks() simply equates: "I want to do
Drew> more than one asynchronous operation inline and here's some syntactic
Drew> sugar using yield as an expression."

Right. I'm just making what's happening underneath a little more forgiving
and consistent in these two edge cases where things go awry.

The main point, to me, is that code changes over time. So you write a
function and use inlineCallbacks decorator for syntactic sugar. Then you,
or someone else, comes along a while later and change a few things. If you
take out the last yield without noticing, or your change throws an
exception before the first yield, then you've broken things and it may not
be clear why. In the first case, you get an exception thrown from deep
inside inlineCallbacks, and many/most people would have some trouble
figuring out what's going on. There's no need for that though.

>> This has the advantage that (barring e.g., a KeyboardInterrupt in the
>> middle of things) you'll *always* get a deferred back when you call an
>> inlineCallbacks decorated function. That deferred might have already
>> called or erred back (corresponding to cases 1 and 2 above).

Drew> You will always get a Deferred  back if the function is a generator.

But if you take out the last deferred, it's no longer a generator. And if
an exception happens before the first yield you don't get a deferred back
because the exception is raised before the inlineCallbacks created function
gets its first value from your generator.

They're just two simple scenarios that can be dealt with more cleanly.  If
it wasn't clear, the inlineCallbacks created function is identical. Your
code is still run identically. There's just a little pre-processing to see
if it even needs to be run.

Drew> That's what unit tests are for :) But I'm not sure how you could
Drew> easily wind up in this scenario considering generators don't allow
Drew> return with an argument.

Right. But a function can just fall off the end, implicitly returning

And I gave two examples, plus simple code, that illustrates how both of the
problems I'm addressing can arise. It's not about having return in a
function. One part is about turning a generator into a non-generator by
removing the final yield. The other's just about introducing an exception
into your code. I've run into both situations, more than once, and in both
had the reaction that inlineCallbacks could have been more accomodating or
helpful in how it behaves.

Fortunately I can just use my own version!  :-)

Sorry if I wasn't being clear enough first time round.


More information about the Twisted-Python mailing list