[Twisted-Python] waiting on any deferred in a list?

Benjamin Rutt rutt.4 at osu.edu
Sun Apr 24 11:37:08 MDT 2016


Hi all,

I've been using a pattern for quite some time to collect a bunch o'
deferreds in a list and then yield on them one at a time.  Here is some
untested pseudocode:

---
@defer.inlineCallbacks
def foo( ls ):
    defs = []

    # get all the deferreds going
    for item in ls:
        d = some_deferred_returning_function( item )
        defs.append( ( item, d ) ) # associate item with deferred

    # process results not as they come in but in the original order
    for ( item, d ) in defs:
        result = yield d
        print 'The result on item', item, 'is', result
---

In this way, I can get good concurrency (by spinning up all deferreds at
once) but also I can iterate on the results using a straightforward
traversal of the list data structure and yield.  This is working great for
me.  But on occasion I'd prefer that I can access the results not in the
order that I originally got, but rather in the order of completion of the
deferreds, so the faster ones come in first, and the slower ones don't hold
up faster ones.  The usual case when I'd want this is when I have "slower"
deferreds near the front, and "faster" deferreds near the back.  I can't
help but think that there's some async data structure out there for me that
can solve this.  I've used DeferredList before, and I can't see that it
does what I want:  it gives you a choice between getting all results only
when they are all done (default behavior), or getting the first one that's
ready (fireOnOneCallback=True).  I want all results, but I want the
processing of the results to unfold as the results come in.

Thanks for any ideas.
-- 
Benjamin Rutt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20160424/f718fb58/attachment.html>


More information about the Twisted-Python mailing list