[Twisted-Python] Re: wired problem with deferreds

[David Bolen]

markus espenhain <me at mocisoft.com> writes:

> im stuck with the following deferred construct - the problem is that cb2
> is called with None instead of the expected argument
>
> from twisted.internet import defer
>
> cd = None
>
> def req1():
>     return defer.Deferred().addCallback(req2)
>
> def req2(a):
>     print 'req2', a
>     global cd
>     if not cd:
>         cd = defer.Deferred().addCallback(req3)
>     return cd
>
> def req3(a):
>     print 'req3', a
>     return a
>
> def cb1(a):
>     print 'cb1', a
>     return a
>
> def cb2(a):
>     print 'cb2', a
>     return a
>
>
> d1 = req1().addCallback(cb1)
> d1.callback('X')
> d2 = req1().addCallback(cb2)
> d2.callback('X')
> cd.callback('S')
>
> output:
>
> req2 X
> req2 X
> req3 S
> cb1 S
> cb2 None # <- ?
>
> req1 fetches a mapping for given arguments - req2 tries to create an
> object + additional resquests by the result of req1 - but these objects
> should only be created once - so far i didn't find an acceptable
> workaround for that - req3 applys some modifications but the problem
> occurs also without the step in req3 
>
> is there something i overlooked so that cb2 isn't called with the
> required arg?

I think it's because you're effectively pausing two callback chains on
the same nested deferred, and it's the continuation processing of that
deferred for the first callback chain that is clearing your result for
the second callback chain.

There are three deferred's involved above, with these initial callbacks:

d1 - Callbacks req2, then cb1
d2 - Callbacks req2 then cb2
cd - Callback req3

Now in each of the d1.callback() and d2.callback() calls, the callback
chain runs the first callback req2(), and then pauses because req2
returns a deferred (cb) that hasn't finished.

It's important to realize that the way the deferred object "pauses" a
callback is to simply insert its own continuation method as a final
callback on the nested deferred.

So, after those two callback() methods, the cd deferred actually has a
callback chain of req3, d1._continue, and d2._continue.

It's also important to note that the internal _continue method of the
deferred object has no return (thus None).  That's where your result
is being lost along the way to cb2.

So, when you finally call cd.callback('S'), I believe you actually
trigger a sequence of events such as:

  req3('S'), returning 'S'
  d1._continue('S'), returning None and unpausing d1
    (recursively)
    cb1('S') due to the next callback in d1's callback chain.
  d2._continue(None), returning None and unpausing d2
    (recursively)
    cb2(None) due to the next callback in d2's callback chain.
  
You'll find that if you temporarily add code to Deferred._continue to
pass through the result it is handed that you see it show up.

I don't know if sharing a nested deferred chain among more than one
outer deferred chain something that can be expected to have sane
behavior.  Then again, I'm not sure it's really a bad thing to have
the Deferred object pass through the result in addition to using it
for itself (e.g., adding a "return result" to the Deferred._continue
method).

You might find that if you need to perform some sort of singleton
action on the underlying operation, that you would be best off
manually handling how that callback affects others than relying on the
default Twisted behavior of nested deferreds.

-- David