[Twisted-Python] Inline callbacks: generating the same Deferred at multiple places

Glyph glyph at twistedmatrix.com
Mon Feb 25 18:23:59 EST 2013

On Feb 23, 2013, at 3:23 AM, Pierre Jaury <pierre at jaury.eu> wrote:

> On 02/22/2013 09:17 PM, Glyph wrote:
>> On Feb 22, 2013, at 8:30 AM, Christopher Armstrong
>> <radix at twistedmatrix.com <mailto:radix at twistedmatrix.com>> wrote:
>>> I think it's a reasonable change to make, and I don't foresee any
>>> problems with it, so I think it's fine to submit a bug about it. But I
>>> do question the architecture that needs to make use of it. I would
>>> probably avoid scenarios like that in my own code.
>> I disagree; the behavior of result consumption is intentional - although
>> it could be better documented.  Changing it would very definitely be
>> incompatible (<http://twistedmatrix.com/trac/wiki/CompatibilityPolicy>);
>> this is possible, of course, if the deprecation/migration is worth it,
>> but the behavior being requested here would be worse in a number of ways.
> Well, some criticism, now I am listening to you.
>> If we re-populated the result, every failed Deferred yielded by an
>> inlineCallbacks function would log its traceback twice: once when the
>> unhandled exception propagated out of the inlineCallbacks function
>> causing its Deferred to fail, and once when the unhandled exception
>> propagated from the yielded Deferred itself, since nothing would have
>> consumed it when that Deferred would be GC'd.
> Well, this simply means that the change is not that simple to make and
> would probably imply some deeper modifications. I can see how this kills
> any hope to get the change merged into anything stable anytime soon however.

The change cannot be made, as such.  The compatibility policy page, linked above, details what would be necessary to add a new API with the behavior you want, but as I think I made clear already, I think that would be a bad idea ;).

>> Speaking of GC, similarly, any large objects in Deferred results
>> processed by inlineCallbacks functions would live longer, and continue
>> participating in any reference cycles they're part of, possibly causing
>> memory leaks, or at least, longer collection times and less favorable
>> memory usage behavior, especially in long-lived processes.
> But.. I definitely agree with this one, which I did not foresee. I
> therefore agree that any change in Deferred objects behavior should
> never enforce references to live longer when not usually required.

Deferred holds enough references to get its job done, no more, no less.  (Well, maybe a little more in the case of a failed Deferred.  But mostly we are talking about a successful Deferred here.)

>> Basically, you can't treat a Deferred as an event broadcaster.  It isn't
>> one.  It's a single-shot representation of an asynchronous result, which
>> a single consumer can consume with its current value, possibly yielding
>> a new value.
> That I figured out already. Thanks for the reminder anyway, I was on the
> path of loosing myself, there :)

OK, good :).

>> Any
>> inlineCallbacks function which wants to express its intent more
>> precisely can do this, instead:
>>    def fork(d):
>>        d2 = Deferred()
>>        def fire(x):
>>            d2.callback(x)
>>            return x
>>        d.addBoth(fire)
>>        return d2
>>    @inlineCallbacks
>>    def foo():
>>        result = yield fork(somethingAsync())
> I will embed this one in my bundle of Twisted utility functions. Thanks!
>> Maybe putting that function in Twisted (this is not the first time it's
>> come up) would be a useful addition.
> Agreed, then.

Some existing implementations:



More information about the Twisted-Python mailing list