[Twisted-Python] Inline callbacks: generating the same Deferred at multiple places

Pierre Jaury pierre at jaury.eu
Sat Feb 23 06:23:56 EST 2013

On 02/22/2013 09:17 PM, Glyph wrote:
> On Feb 22, 2013, at 8:30 AM, Christopher Armstrong
> <radix at twistedmatrix.com <mailto:radix at twistedmatrix.com>> wrote:
>> I think it's a reasonable change to make, and I don't foresee any
>> problems with it, so I think it's fine to submit a bug about it. But I
>> do question the architecture that needs to make use of it. I would
>> probably avoid scenarios like that in my own code.
> I disagree; the behavior of result consumption is intentional - although
> it could be better documented.  Changing it would very definitely be
> incompatible (<http://twistedmatrix.com/trac/wiki/CompatibilityPolicy>);
> this is possible, of course, if the deprecation/migration is worth it,
> but the behavior being requested here would be worse in a number of ways.

Well, some criticism, now I am listening to you.

> If we re-populated the result, every failed Deferred yielded by an
> inlineCallbacks function would log its traceback twice: once when the
> unhandled exception propagated out of the inlineCallbacks function
> causing its Deferred to fail, and once when the unhandled exception
> propagated from the yielded Deferred itself, since nothing would have
> consumed it when that Deferred would be GC'd.

Well, this simply means that the change is not that simple to make and
would probably imply some deeper modifications. I can see how this kills
any hope to get the change merged into anything stable anytime soon however.

> Speaking of GC, similarly, any large objects in Deferred results
> processed by inlineCallbacks functions would live longer, and continue
> participating in any reference cycles they're part of, possibly causing
> memory leaks, or at least, longer collection times and less favorable
> memory usage behavior, especially in long-lived processes.

But.. I definitely agree with this one, which I did not foresee. I
therefore agree that any change in Deferred objects behavior should
never enforce references to live longer when not usually required.

> Basically, you can't treat a Deferred as an event broadcaster.  It isn't
> one.  It's a single-shot representation of an asynchronous result, which
> a single consumer can consume with its current value, possibly yielding
> a new value.

That I figured out already. Thanks for the reminder anyway, I was on the
path of loosing myself, there :)

> Any
> inlineCallbacks function which wants to express its intent more
> precisely can do this, instead:
>     def fork(d):
>         d2 = Deferred()
>         def fire(x):
>             d2.callback(x)
>             return x
>         d.addBoth(fire)
>         return d2
>     @inlineCallbacks
>     def foo():
>         result = yield fork(somethingAsync())

I will embed this one in my bundle of Twisted utility functions. Thanks!

> Maybe putting that function in Twisted (this is not the first time it's
> come up) would be a useful addition.

Agreed, then.

Pierre Jaury <pierre at jaury.eu>
Weblog - http://kaiyou.fr
GPG ID - E804FB60

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: OpenPGP digital signature
Url : http://twistedmatrix.com/pipermail/twisted-python/attachments/20130223/98f27489/attachment.pgp 

More information about the Twisted-Python mailing list