[Twisted-Python] Twisted CallLater: Advanced Usage

Clark C. Evans cce at clarkevans.com
Sun Jul 13 18:43:18 EDT 2003

My last example was wrong, pls disregard. Since hierlistdir does not 
yield a flow.Instruction (flow.Cooperate() or a flow.Stage) it does not
need to be wrapped.   Also, I forgot to have the intermediate 
generator yield flow.Cooperate to return control to the reactor. 
The last example would have been correct if hierlistdir() was
blocking, like an LDAP/SQL query, or a protocol.  Sorry.

    from __future__ import generators
    import os
    from twisted.flow import flow
    def hierlistdir(path):
        """ generator which returns a list of (path, files) 
            tuples for each item in the hierarchy.   Since this
            is a syncronous generator, you could simply do
                print list(hierlistdir("/Some/Path") 
            to verify its operation
        names = os.listdir(path)
        yield (path, names)
        for name in names:
            subpath = os.path.join(path,name)
            if os.path.isdir(subpath):
                for x in hierlistdir(subpath):
                    yield x

    def coopvisit(path, visit, arg):
        """ for each item in the generator, call the visitor
            and then yield Cooperate() so that other requess
            can be handled by the reactor
        for (path,names) in hierlistdir(path):
            yield flow.Cooperate()

    def os_path_walk(path,visit,arg):
        """ provide a nice interface that hides the use of flow,
            and somewhat complies with Moshe's example.
        return flow.Deferred(coopvisit(path,visit,arg))

Although, really, if you were to use flow, you would not
bother to use a vistor pattern, you would just write
your processor as a generator instead.

    def print_if_contains(path, match):
        nMatch = 0
        for (path, names) in hierlistdir(path):
            if match in names:
                nMatch += 1
                print nMatch, path
            yield flow.Cooperate()
    d = flow.Deferred(print_if_contains("/some/path","someFile"))
    d.addCallback(lamda _: reactor.stop())

Where flow "shines" is when the input source blocks, for example,
when reading from a socket (see flow.Protocol), or via an async 
PostgreSQL database, or an LDAP connection.   Or when there are 
ultiple stages of a given flow (a -> b -> c), or when you have
two stages (async PostgreSQL and LDAP queries) which need to
be merged to generate a HTML page...

On Sun, Jul 13, 2003 at 12:24:34PM -0000, Moshe Zadka wrote:
| from __future__ import generators
| def os_path_walk(path, visit, arg):
|     d = defer.Deferred()
|     def walk(path):
|         files = os.listdir(path)
|         visit(arg, path, files)
|         for file in files:
|             yield None
|             if os.path.isdir(file):
|                 for _ in walk(os.path.join(path, file):
|                     pass
|     next = walk(path).next
|     def caller():
|         try:
|             next()
|         except StopIteration:
|             d.callback(None)
|         except:
|             d.errback(sys.exc_info()[1])
|         else:
|             reactor.callLater(0, caller)
|     caller()
|     return d

from __future__ import generators
from twisted.internet import reactor
from twisted.flow import flow
from twisted.python import util

def flowFactorial(n, steps = 10):
    coop = flow.Cooperate()
    acc = 1L
    while True:
        for i in range(steps):
            if 1 == n:
                yield acc
            n, acc = n - 1, acc * n
        yield coop

d = flow.Deferred(flowFactorial(13))
d.addCallback(lambda _: reactor.stop())

More information about the Twisted-Python mailing list