Twisted's
[http://twistedmatrix.com/documents/current/api/twisted.internet.defer.waitForDeferred.html deferred generator]
capability allows you to run iterations in a way that is more intuitive and
similar to what you may be used to seeing in the world of blocking code.

Here's a simple example, courtesy of Alex Levy, that may help you understand
how the concept works:

{{{
#!python

from twisted.internet import defer, reactor

# The reactor is running and we are going to obtain a Deferred "d"

def getSomeDeferred():
    """Some function that returns a Deferred."""
    d = defer.Deferred()
    reactor.callLater(1, d.callback, 'A string that yells "foo!"')
    return d

def anotherDeferred(needle, haystack):
    """Some other function that returns a Deferred."""
    d = defer.Deferred()
    reactor.callLater(1, d.callback, haystack.find(needle))
    return d

@defer.deferredGenerator
def find(needle):
    """A Deferred generator function"""
    print "I am going to find %s in a haystack." % needle
    # After yielding waitForDeferred, our generator
    # will be put on hold for a while.
    wfd = defer.waitForDeferred(getSomeDeferred())
    yield wfd
    # The reactor will call .next(), and we resume here.
    haystack = wfd.getResult()
    print "I got my haystack back from a deferred."
    # We're going to wait for another deferred result.
    wfd = defer.waitForDeferred(anotherDeferred(needle, haystack))
    yield wfd
    # When we get our next result, the procedure resumes here.
    print "I found %s at character %d" % (repr(needle), wfd.getResult())
    reactor.stop()

# We call the deferred generator like any other function and immediately
# get a Deferred that fires when the generator is done.

d = find('foo!')
reactor.run()
}}}

The {{{find}}} function is a generator because it contains {{{yield}}}
statements.  In the blocking world, you would use it to iterate over multiple
values, like this:

{{{
#!python
    for thing in find('foo'):
        print "'%s' is either a needle or a haystack" % thing
}}}

However, waiting for a function to yield something is a form of blocking, which
is a no-no when you're writing asynchronous code.  So we instead wrap the
{{{find}}} function in a {{{defer.deferredGenerator}}}. Now calling it
immediately returns a {{{Deferred}}} that fires when it is __done__ iterating.

The actual stuff that gets done as part of the iteration is incorporated into
the generator function itself. You {{{yield}}} whatever it is you want to be an
iterator of, as usual, with some important differences.

 * You don't yield the actual object, but rather something based on
   a {{{Deferred}}} that will eventually fire with the object.  (If you
   could get the object immediately, there would be no reason to go to all
   this trouble.)

 * To get the actual object from what you yielded (a {{{Deferred}}} that
   has been specially packaged by {{{defer.waitForDeferred}}}), after the
   Twisted event loop has taken time off to go do other stuff, by calling a
   {{{getResult}}} method of the yielded object.

Now let's consider a more detailed example, the deferred generator that types
fake keystrokes, one after the other, in the !WinDictator application.

{{{
#!python

    from twisted.internet import defer

    # The reactor is running and we are inside a class instance that has
    # attributes referencing a "keyer," an "observer," and a "history."

    @defer.deferredGenerator
    def pressAndReleaseKeys(self, keyList):
        """
        This method is decorated with defer.deferredGenerator to immediately
        return a Deferred that fires upon completion of an iteration over
        key combinations, which are supplied as sub-sequences of the 'keyList'
        sequence. The deferred fires with 'True' if all keys were observed to
        have been pressed and reseased, 'False' otherwise.

        The generator sends fake X key events for each key of each
        sub-sequence, a keypress followed by a release, and waits for
        confirmation of each faked key event before proceeding with the next
        one.
        """
        running = True
        
        def gotConfirmation(confirmed):
            running = confirmed
            return confirmed
        
        def keyAndConfirm(func, key):
            d = func(key)
            d.addCallback(self.observer.confirm)
            d.addCallback(gotConfirmation)
            return d
        
        stack = []
        while running and keyList:
            keySequence = keyList.pop(0)
            # Depress each key in order
            for key in keySequence:
                stack.append(key)
                d = keyAndConfirm(self.keyer.pressKey, key)
                wfd = defer.waitForDeferred(d)
                yield wfd
}}}

Here the first {{{waitForDeferred}}} instance is yielded from within a
{{{while}}} loop that is doing a first batch of iterations. Note that we are
yielding a wrapped {{{Deferred}}} that has a fairly complicated callback
chain. Each key of a combination (''e.g.'', "Shift_L + a" = "A") is being
virtually pressed in turn via the hidden machinations of the {{{keyer}}} and
{{{confirm}}} objects.

Processing of the callbacks occurs right inside our generator function, but
it's done whenever the Twisted event loop can get around to it. You may wonder
how that is possible given that Twisted code runs in a single thread.  The
answer lies in the {{{yield}}} statement, which is an invitation for code
outside the generator to do stuff between iterations. See Norman Matloff's fine
[http://heather.cs.ucdavis.edu/~matloff/Python/PyIterGen.pdf tutorial] on
generators for details.

{{{
#!python
                if not wfd.getResult():
                    # If key depression not noted, don't try typing any more of
                    # the text.
                    running = False
                    break
}}}

At this point, we have obtained the value that finally resulted from the calls
to {{{self.keyer.pressKey}}} and {{{self.observer.confirm}}}. The result is a
Boolean indicating that everything went OK in those operations. If it didn't,
we set a flag to prevent further iterations and break out of the current one.

Now comes the next iteration, for releasing the keys we've pressed:

{{{
#!python
            # Release each key in reverse order
            while running and stack:
                key = stack.pop()
                d = keyAndConfirm(self.keyer.releaseKey, key)
                wfd = defer.waitForDeferred(d)
                yield wfd
                if not wfd.getResult():
                    # If key release not noted, don't try typing any more of
                    # the text.
                    running = False
}}}

Again, we yield the {{{Deferred}}} that we've packaged up with
{{{defer.waitForDeferred}}}, let the Twisted event loop do its thing, and get
the deferred result. Again, the result is a Boolean status value and we quit
iterating if there's been a problem.

Now we come to a part of the generator function where we want to deliver a
final result to the caller.  But how do we do that with all this yielding going
on?  How do you even return anything that could be construed as the "result" of
a Python generator, when all you can normally do is iterate over the values it
yields?

Here Twisted's deferred generator adds some functionality to generator
functions.  The last object yielded that is '''not''' wrapped in a call to
{{{defer.waitForDeferred}}}, is used as the deferred result of the
{{{defer.deferredGenerator}}} call itself!

{{{
#!python
        # Final yield of a Boolean, rather than a wfd, is supplied to the
        # deferredGenerator's callback
        yield running
}}}

The {{{running}}} variable holds a status that is {{{True}}} unless made
{{{False}}} by any of the key press and release operations. That's also the
status of the entire key generator operation, and whatever function called it
gets that as the deferred result.

A method in the same class as {{{pressAndReleaseKeys}}} is an example of such a
function:

{{{
#!python
    def backspace(self, N):
        """
        Deletes backwards I{N} characters using the C{BackSpace} keysym.
        """
        if self.typingEnabled():
            self.history.backspace(N)
            keyList = [("BackSpace",)] * N
            d = self.pressAndReleaseKeys(keyList)
        else:
            # Typing isn't enabled, just return a Deferred that immediately
            # fires with False typing-failed status
            d = defer.succeed(False)
        return d
}}}

The method calls {{{self.pressAndReleaseKeys()}}} and __immediately__ gets a
{{{Deferred}}} as the result even while the generator function is hammering out
backspaces one by one. The {{{backspace}}} method, by the way, is used for
making corrections to dictated text, which works surprisingly well.

Here's another example of a method that uses our deferred generator, again in
the same class:

{{{
#!python
    def insert(self, text):
        """
        Parses the supplied I{text} and sends chunks to the appropriate methods
        of the keyer, returning C{True} if entry of all text was noted and
        C{False} if not.
        """
        if self.typingEnabled:
            text = self.contextAdjust(text)
            keyList = self.keyCoder.textToKeys(text)
            d = self.pressAndReleaseKeys(keyList)
        else:
            # Typing isn't enabled, just return a Deferred that immediately
            # fires with False typing-failed status
            d = defer.succeed(False)
        return d
}}}