# Ticket #1009: defgen.txt

File defgen.txt, 9.5 KB (added by edsuom, 13 years ago)
Line
1Twisted's
2[http://twistedmatrix.com/documents/current/api/twisted.internet.defer.waitForDeferred.html deferred generator]
3capability allows you to run iterations in a way that is more intuitive and
4similar to what you may be used to seeing in the world of blocking code.
5
6Here's a simple example, courtesy of Alex Levy, that may help you understand
7how the concept works:
8
9{{{
10#!python
11
12    from twisted.internet import defer
13
14    # The reactor is running and we are going to obtain a Deferred "d"
15
16    def getSomeDeferred():
17         """Some function that returns a Deferred."""
18         d = defer.Deferred()
19         reactor.callLater(2, d.callback, 'A string that yells "foo!"')
20         return d
21
22    def anotherDeferred(needle, haystack):
23         """Some other function that returns a Deferred."""
24         d = defer.Deferred()
25         reactor.callLater(2, d.callback, haystack.find(needle))
26         return d
27
28    @defer.deferredGenerator
29    def find(needle):
30         """A Deferred generator function"""
31         print "I am going to find %s in a haystack." % needle
32         # After yielding waitForDeferred, our generator
33         # will be put on hold for a while.
34         wfd = defer.waitForDeferred(getSomeDeferred())
35         yield wfd
36         # The reactor will call .next(), and we resume here.
37         haystack = wfd.getResult()
38         print "I got my haystack back from a deferred."
39         # We're going to wait for another deferred result.
40         wfd = defer.waitForDeferred(anotherDeferred(needle, haystack))
41         yield wfd
42         # When we get our next result, the procedure resumes here.
43         print "I found %s at character %d" % (repr(needle), d.getResult())
44
45    # We call the deferred generator like any other function and immediately
46    # get a Deferred that fires when the generator is done.
47
48    d = find('foo!')
49}}}
50
51The {{{find}}} function is a generator because it contains {{{yield}}}
52statements.  In the blocking world, you would use it to iterate over multiple
53values, like this:
54
55{{{
56#!python
57    for thing in find('foo'):
58        print "'%s' is either a needle or a haystack" % thing
59}}}
60
61However, waiting for a function to yield something is a form of blocking, which
62is a no-no when you're writing asynchronous code.  So we instead wrap the
63{{{find}}} function in a {{{defer.deferredGenerator}}}. Now calling it
64immediately returns a {{{Deferred}}} that fires when it is __done__ iterating.
65
66The actual stuff that gets done as part of the iteration is incorporated into
67the generator function itself. You {{{yield}}} whatever it is you want to be an
68iterator of, as usual, with some important differences.
69
70 * You don't yield the actual object, but rather something based on
71   a {{{Deferred}}} that will eventually fire with the object.  (If you
72   could get the object immediately, there would be no reason to go to all
73   this trouble.)
74
75 * To get the actual object from what you yielded (a {{{Deferred}}} that
76   has been specially packaged by {{{defer.waitForDeferred}}}), after the
77   Twisted event loop has taken time off to go do other stuff, by calling a
78   {{{getResult}}} method of the yielded object.
79
80Now let's consider a more detailed example, the deferred generator that types
81fake keystrokes, one after the other, in the !WinDictator application.
82
83{{{
84#!python
85
86    from twisted.internet import defer
87
88    # The reactor is running and we are inside a class instance that has
89    # attributes referencing a "keyer," an "observer," and a "history."
90
91    @defer.deferredGenerator
92    def pressAndReleaseKeys(self, keyList):
93        """
94        This method is decorated with defer.deferredGenerator to immediately
95        return a Deferred that fires upon completion of an iteration over
96        key combinations, which are supplied as sub-sequences of the 'keyList'
97        sequence. The deferred fires with 'True' if all keys were observed to
98        have been pressed and reseased, 'False' otherwise.
99
100        The generator sends fake X key events for each key of each
101        sub-sequence, a keypress followed by a release, and waits for
102        confirmation of each faked key event before proceeding with the next
103        one.
104        """
105        running = True
106
107        def gotConfirmation(confirmed):
108            running = confirmed
109            return confirmed
110
111        def keyAndConfirm(func, key):
112            d = func(key)
113            d.addCallback(self.observer.confirm)
114            d.addCallback(gotConfirmation)
115            return d
116
117        stack = []
118        while running and keyList:
119            keySequence = keyList.pop(0)
120            # Depress each key in order
121            for key in keySequence:
122                stack.append(key)
123                d = keyAndConfirm(self.keyer.pressKey, key)
124                wfd = defer.waitForDeferred(d)
125                yield wfd
126}}}
127
128Here the first {{{waitForDeferred}}} instance is yielded from within a
129{{{while}}} loop that is doing a first batch of iterations. Note that we are
130yielding a wrapped {{{Deferred}}} that has a fairly complicated callback
131chain. Each key of a combination (''e.g.'', "Shift_L + a" = "A") is being
132virtually pressed in turn via the hidden machinations of the {{{keyer}}} and
133{{{confirm}}} objects.
134
135Processing of the callbacks occurs right inside our generator function, but
136it's done whenever the Twisted event loop can get around to it. You may wonder
137how that is possible given that Twisted code runs in a single thread.  The
138answer lies in the {{{yield}}} statement, which is an invitation for code
139outside the generator to do stuff between iterations. See Norman Matloff's fine
140[http://heather.cs.ucdavis.edu/~matloff/Python/PyIterGen.pdf tutorial] on
141generators for details.
142
143{{{
144#!python
145                if not wfd.getResult():
146                    # If key depression not noted, don't try typing any more of
147                    # the text.
148                    running = False
149                    break
150}}}
151
152At this point, we have obtained the value that finally resulted from the calls
153to {{{self.keyer.pressKey}}} and {{{self.observer.confirm}}}. The result is a
154Boolean indicating that everything went OK in those operations. If it didn't,
155we set a flag to prevent further iterations and break out of the current one.
156
157Now comes the next iteration, for releasing the keys we've pressed:
158
159{{{
160#!python
161            # Release each key in reverse order
162            while running and stack:
163                key = stack.pop()
164                d = keyAndConfirm(self.keyer.releaseKey, key)
165                wfd = defer.waitForDeferred(d)
166                yield wfd
167                if not wfd.getResult():
168                    # If key release not noted, don't try typing any more of
169                    # the text.
170                    running = False
171}}}
172
173Again, we yield the {{{Deferred}}} that we've packaged up with
174{{{defer.waitForDeferred}}}, let the Twisted event loop do its thing, and get
175the deferred result. Again, the result is a Boolean status value and we quit
176iterating if there's been a problem.
177
178Now we come to a part of the generator function where we want to deliver a
179final result to the caller.  But how do we do that with all this yielding going
180on?  How do you even return anything that could be construed as the "result" of
181a Python generator, when all you can normally do is iterate over the values it
182yields?
183
184Here Twisted's deferred generator adds some functionality to generator
185functions.  The last object yielded that is '''not''' wrapped in a call to
186{{{defer.waitForDeferred}}}, is used as the deferred result of the
187{{{defer.deferredGenerator}}} call itself!
188
189{{{
190#!python
191        # Final yield of a Boolean, rather than a wfd, is supplied to the
192        # deferredGenerator's callback
193        yield running
194}}}
195
196The {{{running}}} variable holds a status that is {{{True}}} unless made
197{{{False}}} by any of the key press and release operations. That's also the
198status of the entire key generator operation, and whatever function called it
199gets that as the deferred result.
200
201A method in the same class as {{{pressAndReleaseKeys}}} is an example of such a
202function:
203
204{{{
205#!python
206    def backspace(self, N):
207        """
208        Deletes backwards I{N} characters using the C{BackSpace} keysym.
209        """
210        if self.typingEnabled():
211            self.history.backspace(N)
212            keyList = [("BackSpace",)] * N
213            d = self.pressAndReleaseKeys(keyList)
214        else:
215            # Typing isn't enabled, just return a Deferred that immediately
216            # fires with False typing-failed status
217            d = defer.succeed(False)
218        return d
219}}}
220
221The method calls {{{self.pressAndReleaseKeys()}}} and __immediately__ gets a
222{{{Deferred}}} as the result even while the generator function is hammering out
223backspaces one by one. The {{{backspace}}} method, by the way, is used for
224making corrections to dictated text, which works surprisingly well.
225
226Here's another example of a method that uses our deferred generator, again in
227the same class:
228
229{{{
230#!python
231    def insert(self, text):
232        """
233        Parses the supplied I{text} and sends chunks to the appropriate methods
234        of the keyer, returning C{True} if entry of all text was noted and
235        C{False} if not.
236        """
237        if self.typingEnabled:
238            text = self.contextAdjust(text)
239            keyList = self.keyCoder.textToKeys(text)
240            d = self.pressAndReleaseKeys(keyList)
241        else:
242            # Typing isn't enabled, just return a Deferred that immediately
243            # fires with False typing-failed status
244            d = defer.succeed(False)
245        return d
246}}}
247