[Twisted-Python] Sequential use of asynchronous calls

Maarten ter Huurne maarten at treewalker.org
Sat May 26 08:22:15 MDT 2007


Hi,

Sometimes I want to use several asynchronous calls in a fixed sequence. For 
example, a web application might:
- authenticate the user
- fetch info from a database
- present the result

Implementing this using Deferreds and separate callback+errback functions has 
the disadvantage that the sequence itself is not easy to recognise anymore, 
as it gets spread out over multiple functions.

So I got creative with the new generator features of Python 2.5 and came up 
with a decorator named "sequential", which can be applied to generator 
functions. It consumes Deferreds that are yielded by the generator and sends 
back the result when it becomes available, or raises an Exception in the 
generator if the deferred action fails.

The decorated function returns a Deferred itself, which is fired upon 
completion of the sequence. In particular, this allows nesting sequences 
inside sequences.

This is an example of a program using it, it is an elaborated version of the 
first example from the Deferred Reference:

===
from twisted.internet import defer, reactor
from twisted.python import log

from sequential import sequential

def getDummyData(x):
    d = defer.Deferred()
    if x < 0:
        reactor.callLater(1, d.errback, ValueError('negative value: %d' % x))
    else:
        reactor.callLater(1, d.callback, x * 3)
    return d

@sequential
def work():
    print (yield getDummyData(3))
    print (yield getDummyData(4))
    print (yield 'immediate')
    print (yield getDummyData(6))
    try:
        print (yield getDummyData(-7))
    except ValueError, e:
        print 'failed:', e

@sequential
def main(message):
    print message, 'once...'
    yield work()
    print message, 'twice...'
    yield work()

def done(result):
    reactor.stop()

def failed(fail):
    log.err(fail)
    reactor.stop()

d = main('going')
d.addCallback(done)
d.addErrback(failed)

reactor.run()
===

And here is the implementation of the "sequential" module:

===
from twisted.internet import defer
from twisted.python import failure

from functools import wraps
from compiler.consts import CO_GENERATOR

class _SequentialCaller(object):
    '''Repeatedly reads a Deferred from a generator and feeds it back the 
    result when it becomes available.
    '''

    def __init__(self, gen):
        self.gen = gen
        self.deferred = defer.Deferred()

    def start(self):
        self.next(None)
        return self.deferred

    def next(self, result):
        while True:
            try:
                if isinstance(result, failure.Failure):
                    traceback = result.getTracebackObject() \
                        if hasattr(result, 'getTracebackObject') else None
                    d = self.gen.throw(
                        result.type, result.getErrorMessage(), traceback
                        )
                else:
                    d = self.gen.send(result)
            except StopIteration:
                self.deferred.callback(None)
                return
            except StandardError:
                self.deferred.errback(failure.Failure())
                return
            if isinstance(d, defer.Deferred):
                d.addCallback(lambda result: self.next(result))
                d.addErrback(lambda fail: self.next(fail))
                return
            else:
                # Allow non-deferred values as well: for some Twisted calls,
                # you don't know whether the result will be deferred or not.
                result = d

def sequential(f):
    if not (f.func_code.co_flags & CO_GENERATOR):
        raise TypeError('function "%s" is not a generator' % f.__name__)
    @wraps(f)
    def wrapper(*args, **kvArgs):
        return _SequentialCaller(f(*args, **kvArgs)).start()
    return wrapper
===

I'd like some feedback on this:
- would you consider this useful?
- is the interface right or can it be improved?
- is the implementation correct? (the example scenario doesn't test the error 
path extensively, so there might be problems there)
- is the use of Failure.getTracebackObject correct? (the version of Twisted 
installed on my machine does not have it yet, I only read about it in the 
sources on the API documentation site)
- the "compiler.consts" module is not documented in the Python Library 
Reference, does that mean it should not be used or did they forget to 
document it?
- anything else you'd like to say about it

Is there already something like this in Twisted or one of the toolkits built 
on Twisted? I took a quick look at the "flow" modules, but that seems like a 
more generic and flexible, but also more complex, approach.

If it would be a useful addition to Twisted or a Twisted-based toolkit, I'm 
willing to improve the documentation and write test cases.

Bye,
		Maarten
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: </pipermail/twisted-python/attachments/20070526/55a902dd/attachment.sig>


More information about the Twisted-Python mailing list