wiki:DeferredGenerator

Version 2 (modified by yang, 8 years ago) (diff)

minor fixes in code snippet

Twisted's deferred generator capability allows you to run iterations in a way that is more intuitive and similar to what you may be used to seeing in the world of blocking code.

Here's a simple example, courtesy of Alex Levy, that may help you understand how the concept works:

from twisted.internet import defer, reactor

# The reactor is running and we are going to obtain a Deferred "d"

def getSomeDeferred():
    """Some function that returns a Deferred."""
    d = defer.Deferred()
    reactor.callLater(1, d.callback, 'A string that yells "foo!"')
    return d

def anotherDeferred(needle, haystack):
    """Some other function that returns a Deferred."""
    d = defer.Deferred()
    reactor.callLater(1, d.callback, haystack.find(needle))
    return d

@defer.deferredGenerator
def find(needle):
    """A Deferred generator function"""
    print "I am going to find %s in a haystack." % needle
    # After yielding waitForDeferred, our generator
    # will be put on hold for a while.
    wfd = defer.waitForDeferred(getSomeDeferred())
    yield wfd
    # The reactor will call .next(), and we resume here.
    haystack = wfd.getResult()
    print "I got my haystack back from a deferred."
    # We're going to wait for another deferred result.
    wfd = defer.waitForDeferred(anotherDeferred(needle, haystack))
    yield wfd
    # When we get our next result, the procedure resumes here.
    print "I found %s at character %d" % (repr(needle), wfd.getResult())
    reactor.stop()

# We call the deferred generator like any other function and immediately
# get a Deferred that fires when the generator is done.

d = find('foo!')
reactor.run()

The find function is a generator because it contains yield statements. In the blocking world, you would use it to iterate over multiple values, like this:

    for thing in find('foo'):
        print "'%s' is either a needle or a haystack" % thing

However, waiting for a function to yield something is a form of blocking, which is a no-no when you're writing asynchronous code. So we instead wrap the find function in a defer.deferredGenerator. Now calling it immediately returns a Deferred that fires when it is done iterating.

The actual stuff that gets done as part of the iteration is incorporated into the generator function itself. You yield whatever it is you want to be an iterator of, as usual, with some important differences.

  • You don't yield the actual object, but rather something based on a Deferred that will eventually fire with the object. (If you could get the object immediately, there would be no reason to go to all this trouble.)
  • To get the actual object from what you yielded (a Deferred that has been specially packaged by defer.waitForDeferred), after the Twisted event loop has taken time off to go do other stuff, by calling a getResult method of the yielded object.

Now let's consider a more detailed example, the deferred generator that types fake keystrokes, one after the other, in the WinDictator application.

    from twisted.internet import defer

    # The reactor is running and we are inside a class instance that has
    # attributes referencing a "keyer," an "observer," and a "history."

    @defer.deferredGenerator
    def pressAndReleaseKeys(self, keyList):
        """
        This method is decorated with defer.deferredGenerator to immediately
        return a Deferred that fires upon completion of an iteration over
        key combinations, which are supplied as sub-sequences of the 'keyList'
        sequence. The deferred fires with 'True' if all keys were observed to
        have been pressed and reseased, 'False' otherwise.

        The generator sends fake X key events for each key of each
        sub-sequence, a keypress followed by a release, and waits for
        confirmation of each faked key event before proceeding with the next
        one.
        """
        running = True
        
        def gotConfirmation(confirmed):
            running = confirmed
            return confirmed
        
        def keyAndConfirm(func, key):
            d = func(key)
            d.addCallback(self.observer.confirm)
            d.addCallback(gotConfirmation)
            return d
        
        stack = []
        while running and keyList:
            keySequence = keyList.pop(0)
            # Depress each key in order
            for key in keySequence:
                stack.append(key)
                d = keyAndConfirm(self.keyer.pressKey, key)
                wfd = defer.waitForDeferred(d)
                yield wfd

Here the first waitForDeferred instance is yielded from within a while loop that is doing a first batch of iterations. Note that we are yielding a wrapped Deferred that has a fairly complicated callback chain. Each key of a combination (e.g., "Shift_L + a" = "A") is being virtually pressed in turn via the hidden machinations of the keyer and confirm objects.

Processing of the callbacks occurs right inside our generator function, but it's done whenever the Twisted event loop can get around to it. You may wonder how that is possible given that Twisted code runs in a single thread. The answer lies in the yield statement, which is an invitation for code outside the generator to do stuff between iterations. See Norman Matloff's fine tutorial on generators for details.

                if not wfd.getResult():
                    # If key depression not noted, don't try typing any more of
                    # the text.
                    running = False
                    break

At this point, we have obtained the value that finally resulted from the calls to self.keyer.pressKey and self.observer.confirm. The result is a Boolean indicating that everything went OK in those operations. If it didn't, we set a flag to prevent further iterations and break out of the current one.

Now comes the next iteration, for releasing the keys we've pressed:

            # Release each key in reverse order
            while running and stack:
                key = stack.pop()
                d = keyAndConfirm(self.keyer.releaseKey, key)
                wfd = defer.waitForDeferred(d)
                yield wfd
                if not wfd.getResult():
                    # If key release not noted, don't try typing any more of
                    # the text.
                    running = False

Again, we yield the Deferred that we've packaged up with defer.waitForDeferred, let the Twisted event loop do its thing, and get the deferred result. Again, the result is a Boolean status value and we quit iterating if there's been a problem.

Now we come to a part of the generator function where we want to deliver a final result to the caller. But how do we do that with all this yielding going on? How do you even return anything that could be construed as the "result" of a Python generator, when all you can normally do is iterate over the values it yields?

Here Twisted's deferred generator adds some functionality to generator functions. The last object yielded that is not wrapped in a call to defer.waitForDeferred, is used as the deferred result of the defer.deferredGenerator call itself!

        # Final yield of a Boolean, rather than a wfd, is supplied to the
        # deferredGenerator's callback
        yield running

The running variable holds a status that is True unless made False by any of the key press and release operations. That's also the status of the entire key generator operation, and whatever function called it gets that as the deferred result.

A method in the same class as pressAndReleaseKeys is an example of such a function:

    def backspace(self, N):
        """
        Deletes backwards I{N} characters using the C{BackSpace} keysym.
        """
        if self.typingEnabled():
            self.history.backspace(N)
            keyList = [("BackSpace",)] * N
            d = self.pressAndReleaseKeys(keyList)
        else:
            # Typing isn't enabled, just return a Deferred that immediately
            # fires with False typing-failed status
            d = defer.succeed(False)
        return d

The method calls self.pressAndReleaseKeys() and immediately gets a Deferred as the result even while the generator function is hammering out backspaces one by one. The backspace method, by the way, is used for making corrections to dictated text, which works surprisingly well.

Here's another example of a method that uses our deferred generator, again in the same class:

    def insert(self, text):
        """
        Parses the supplied I{text} and sends chunks to the appropriate methods
        of the keyer, returning C{True} if entry of all text was noted and
        C{False} if not.
        """
        if self.typingEnabled:
            text = self.contextAdjust(text)
            keyList = self.keyCoder.textToKeys(text)
            d = self.pressAndReleaseKeys(keyList)
        else:
            # Typing isn't enabled, just return a Deferred that immediately
            # fires with False typing-failed status
            d = defer.succeed(False)
        return d