[Twisted-Python] Re: DeferredList.addDeferred behaviour

[David Bolen]

Andrew Bennetts <andrew-twisted at puzzling.org> writes:

> It's not clear to me what the right behaviour here is -- if a DeferredList
> has some number of Deferreds that have all been called, then of course the
> DeferredList should be marked as called, too.  But if you then add an
> uncalled Deferred to it, that sounds like a bug.  I'd be inclined to make
> DeferredList.addDeferred raise an exception in this case -- "In the face of
> ambiguity, refuse the temptation to guess." (although perhaps a warning
> would be better for a release or two, for backwards compatibility)

Well, it's not clear to me that having a mixture of called and
uncalled in a DeferredList is necessarily inconsistent - at least not
until after the DeferredList has actually has had its first callback
assigned.  After all, even if you insert just uncalled Deferreds
initially, during the course of normal events you'll end up with a
mixture as some of the Deferreds fire and others haven't yet.

I do agree that adding an uncalled Deferred to a DeferredList after
that DeferredList has really fired (called actual callbacks) is a
problem.  I think the HowTo warns against this case though, but it
says to check via the called attribute, which of course this thread
seems to point out as potentially a problem.

> Can anyone offer reasons why addDeferred should stay?  Donovan -- CVS says
> you added it, can you remember why?  Is anyone using it?

I was until I saw this thread and figured out it probably wasn't going
to hold up the way I expected.  It was working for me now but that's
because the existing deferrable objects were returning Deferreds
already called (mostly through defer.succeed and defer.fail) as
placeholders for eventual remote versions, and thus when I was finally
adding the callbacks/errbacks to the DeferredList they were running
synchronously just fine.  But it looks like the problem that started
this thread would break that code when I started having uncalled
results as part of the DeferredList since the callbacks would still
get called as soon as they were added.

As a use case, my scenario is an upper level package function (that
itself returns a deferrable interface) that performs a function across
several target objects as part of its operation - sort of a deferrable
version of map.  I want to use the DeferredList to aggregate/process
the results into a single result, so I want to completely absorb the
individual deferreds.  It seemed to me that this should have been a good
fit for DeferredList.

Because DeferredList itself is just a callback and won't terminate the
original deferred chain (in particular, it doesn't stop the errback),
I also need to add a specific errback to each individual Deferred to
absorb any errors.  Without addDeferred I can't do this in a single loop.

For example, assuming that "func" is a deferrable method on a number
of objects held in an iterable "objs":

With addDeferred (the code I used to have):

    dl = defer.DeferredList([])
    for obj in objs:
        d = obj.func()
        dl.addDeferred(d)
        d.addErrback(lambda _:None)
    dl.addCallback(processResults)
    return dl

without addDeferred (the code I have now):

    deferreds = []
    for obj in objs:
        d = obj.func()
        deferreds.append(d)
    dl = defer.DeferredList([])

    for d in deferreds:
        d.addErrback(lambda _:None)

    dl.addCallback(processResults)
    return dl

Without addDeferred I need to aggregate the underlying deferreds
separately, but more importantly, I need a completely separate loop to
install the suppression errbacks, because that has to be done _after_
the Deferreds are stuck into the DeferredList or else that suppression
will stop the DeferredList from seeing the error.

Maybe not an extreme case, but at the time of writing it I thought
that this was specifically the sort of case that addDeferred was
perfect for.

Although if DeferredList were to grow a way to specify that it should
suppress errors once it has gathered them, I'd probably be just as
happy as having addDeferred fixed, since the separate loop for error
suppression bugs me more than the need to separately aggregate the
Deferreds together before creating the DeferredList.  Actually
suppressing errors might make me even happier, since needing to
suppress the errors separately (noted in the API documentation, but
not the HowTo), while making perfect sense to me now, took a while to
catch on to initially.

-- David