[Twisted-Python] Sequential use of asynchronous calls
Maarten ter Huurne
maarten at treewalker.org
Sat May 26 08:22:15 MDT 2007
Hi,
Sometimes I want to use several asynchronous calls in a fixed sequence. For
example, a web application might:
- authenticate the user
- fetch info from a database
- present the result
Implementing this using Deferreds and separate callback+errback functions has
the disadvantage that the sequence itself is not easy to recognise anymore,
as it gets spread out over multiple functions.
So I got creative with the new generator features of Python 2.5 and came up
with a decorator named "sequential", which can be applied to generator
functions. It consumes Deferreds that are yielded by the generator and sends
back the result when it becomes available, or raises an Exception in the
generator if the deferred action fails.
The decorated function returns a Deferred itself, which is fired upon
completion of the sequence. In particular, this allows nesting sequences
inside sequences.
This is an example of a program using it, it is an elaborated version of the
first example from the Deferred Reference:
===
from twisted.internet import defer, reactor
from twisted.python import log
from sequential import sequential
def getDummyData(x):
d = defer.Deferred()
if x < 0:
reactor.callLater(1, d.errback, ValueError('negative value: %d' % x))
else:
reactor.callLater(1, d.callback, x * 3)
return d
@sequential
def work():
print (yield getDummyData(3))
print (yield getDummyData(4))
print (yield 'immediate')
print (yield getDummyData(6))
try:
print (yield getDummyData(-7))
except ValueError, e:
print 'failed:', e
@sequential
def main(message):
print message, 'once...'
yield work()
print message, 'twice...'
yield work()
def done(result):
reactor.stop()
def failed(fail):
log.err(fail)
reactor.stop()
d = main('going')
d.addCallback(done)
d.addErrback(failed)
reactor.run()
===
And here is the implementation of the "sequential" module:
===
from twisted.internet import defer
from twisted.python import failure
from functools import wraps
from compiler.consts import CO_GENERATOR
class _SequentialCaller(object):
'''Repeatedly reads a Deferred from a generator and feeds it back the
result when it becomes available.
'''
def __init__(self, gen):
self.gen = gen
self.deferred = defer.Deferred()
def start(self):
self.next(None)
return self.deferred
def next(self, result):
while True:
try:
if isinstance(result, failure.Failure):
traceback = result.getTracebackObject() \
if hasattr(result, 'getTracebackObject') else None
d = self.gen.throw(
result.type, result.getErrorMessage(), traceback
)
else:
d = self.gen.send(result)
except StopIteration:
self.deferred.callback(None)
return
except StandardError:
self.deferred.errback(failure.Failure())
return
if isinstance(d, defer.Deferred):
d.addCallback(lambda result: self.next(result))
d.addErrback(lambda fail: self.next(fail))
return
else:
# Allow non-deferred values as well: for some Twisted calls,
# you don't know whether the result will be deferred or not.
result = d
def sequential(f):
if not (f.func_code.co_flags & CO_GENERATOR):
raise TypeError('function "%s" is not a generator' % f.__name__)
@wraps(f)
def wrapper(*args, **kvArgs):
return _SequentialCaller(f(*args, **kvArgs)).start()
return wrapper
===
I'd like some feedback on this:
- would you consider this useful?
- is the interface right or can it be improved?
- is the implementation correct? (the example scenario doesn't test the error
path extensively, so there might be problems there)
- is the use of Failure.getTracebackObject correct? (the version of Twisted
installed on my machine does not have it yet, I only read about it in the
sources on the API documentation site)
- the "compiler.consts" module is not documented in the Python Library
Reference, does that mean it should not be used or did they forget to
document it?
- anything else you'd like to say about it
Is there already something like this in Twisted or one of the toolkits built
on Twisted? I took a quick look at the "flow" modules, but that seems like a
more generic and flexible, but also more complex, approach.
If it would be a useful addition to Twisted or a Twisted-based toolkit, I'm
willing to improve the documentation and write test cases.
Bye,
Maarten
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: </pipermail/twisted-python/attachments/20070526/55a902dd/attachment.sig>
More information about the Twisted-Python
mailing list