[Twisted-Python] Sequential use of asynchronous calls

[Ed Suominen]

http://foss.eepatents.com/api/twisted-goodies/taskqueue.base.TaskQueue.html

Maarten ter Huurne wrote:
> Hi,
> 
> Sometimes I want to use several asynchronous calls in a fixed sequence. For 
> example, a web application might:
> - authenticate the user
> - fetch info from a database
> - present the result
> 
> Implementing this using Deferreds and separate callback+errback functions has 
> the disadvantage that the sequence itself is not easy to recognise anymore, 
> as it gets spread out over multiple functions.
> 
> So I got creative with the new generator features of Python 2.5 and came up 
> with a decorator named "sequential", which can be applied to generator 
> functions. It consumes Deferreds that are yielded by the generator and sends 
> back the result when it becomes available, or raises an Exception in the 
> generator if the deferred action fails.
> 
> The decorated function returns a Deferred itself, which is fired upon 
> completion of the sequence. In particular, this allows nesting sequences 
> inside sequences.
> 
> This is an example of a program using it, it is an elaborated version of the 
> first example from the Deferred Reference:
> 
> ===
> from twisted.internet import defer, reactor
> from twisted.python import log
> 
> from sequential import sequential
> 
> def getDummyData(x):
>     d = defer.Deferred()
>     if x < 0:
>         reactor.callLater(1, d.errback, ValueError('negative value: %d' % x))
>     else:
>         reactor.callLater(1, d.callback, x * 3)
>     return d
> 
> @sequential
> def work():
>     print (yield getDummyData(3))
>     print (yield getDummyData(4))
>     print (yield 'immediate')
>     print (yield getDummyData(6))
>     try:
>         print (yield getDummyData(-7))
>     except ValueError, e:
>         print 'failed:', e
> 
> @sequential
> def main(message):
>     print message, 'once...'
>     yield work()
>     print message, 'twice...'
>     yield work()
> 
> def done(result):
>     reactor.stop()
> 
> def failed(fail):
>     log.err(fail)
>     reactor.stop()
> 
> d = main('going')
> d.addCallback(done)
> d.addErrback(failed)
> 
> reactor.run()
> ===
> 
> And here is the implementation of the "sequential" module:
> 
> ===
> from twisted.internet import defer
> from twisted.python import failure
> 
> from functools import wraps
> from compiler.consts import CO_GENERATOR
> 
> class _SequentialCaller(object):
>     '''Repeatedly reads a Deferred from a generator and feeds it back the 
>     result when it becomes available.
>     '''
> 
>     def __init__(self, gen):
>         self.gen = gen
>         self.deferred = defer.Deferred()
> 
>     def start(self):
>         self.next(None)
>         return self.deferred
> 
>     def next(self, result):
>         while True:
>             try:
>                 if isinstance(result, failure.Failure):
>                     traceback = result.getTracebackObject() \
>                         if hasattr(result, 'getTracebackObject') else None
>                     d = self.gen.throw(
>                         result.type, result.getErrorMessage(), traceback
>                         )
>                 else:
>                     d = self.gen.send(result)
>             except StopIteration:
>                 self.deferred.callback(None)
>                 return
>             except StandardError:
>                 self.deferred.errback(failure.Failure())
>                 return
>             if isinstance(d, defer.Deferred):
>                 d.addCallback(lambda result: self.next(result))
>                 d.addErrback(lambda fail: self.next(fail))
>                 return
>             else:
>                 # Allow non-deferred values as well: for some Twisted calls,
>                 # you don't know whether the result will be deferred or not.
>                 result = d
> 
> def sequential(f):
>     if not (f.func_code.co_flags & CO_GENERATOR):
>         raise TypeError('function "%s" is not a generator' % f.__name__)
>     @wraps(f)
>     def wrapper(*args, **kvArgs):
>         return _SequentialCaller(f(*args, **kvArgs)).start()
>     return wrapper
> ===
> 
> I'd like some feedback on this:
> - would you consider this useful?
> - is the interface right or can it be improved?
> - is the implementation correct? (the example scenario doesn't test the error 
> path extensively, so there might be problems there)
> - is the use of Failure.getTracebackObject correct? (the version of Twisted 
> installed on my machine does not have it yet, I only read about it in the 
> sources on the API documentation site)
> - the "compiler.consts" module is not documented in the Python Library 
> Reference, does that mean it should not be used or did they forget to 
> document it?
> - anything else you'd like to say about it
> 
> Is there already something like this in Twisted or one of the toolkits built 
> on Twisted? I took a quick look at the "flow" modules, but that seems like a 
> more generic and flexible, but also more complex, approach.
> 
> If it would be a useful addition to Twisted or a Twisted-based toolkit, I'm 
> willing to improve the documentation and write test cases.
> 
> Bye,
> 		Maarten
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python