[Twisted-Python] Twisted CallLater: Advanced Usage

[Clark C. Evans]

My last example was wrong, pls disregard. Since hierlistdir does not 
yield a flow.Instruction (flow.Cooperate() or a flow.Stage) it does not
need to be wrapped.   Also, I forgot to have the intermediate 
generator yield flow.Cooperate to return control to the reactor. 
The last example would have been correct if hierlistdir() was
blocking, like an LDAP/SQL query, or a protocol.  Sorry.

    from __future__ import generators
    import os
    from twisted.flow import flow
    
    def hierlistdir(path):
        """ generator which returns a list of (path, files) 
            tuples for each item in the hierarchy.   Since this
            is a syncronous generator, you could simply do
                print list(hierlistdir("/Some/Path") 
            to verify its operation
        """
        names = os.listdir(path)
        yield (path, names)
        for name in names:
            subpath = os.path.join(path,name)
            if os.path.isdir(subpath):
                for x in hierlistdir(subpath):
                    yield x

    def coopvisit(path, visit, arg):
        """ for each item in the generator, call the visitor
            and then yield Cooperate() so that other requess
            can be handled by the reactor
        """
        for (path,names) in hierlistdir(path):
            visit(arg,path,names)
            yield flow.Cooperate()

    def os_path_walk(path,visit,arg):
        """ provide a nice interface that hides the use of flow,
            and somewhat complies with Moshe's example.
        """
        return flow.Deferred(coopvisit(path,visit,arg))

Although, really, if you were to use flow, you would not
bother to use a vistor pattern, you would just write
your processor as a generator instead.

    def print_if_contains(path, match):
        nMatch = 0
        for (path, names) in hierlistdir(path):
            if match in names:
                nMatch += 1
                print nMatch, path
            yield flow.Cooperate()
     
    d = flow.Deferred(print_if_contains("/some/path","someFile"))
    d.addCallback(lamda _: reactor.stop())

Where flow "shines" is when the input source blocks, for example,
when reading from a socket (see flow.Protocol), or via an async 
PostgreSQL database, or an LDAP connection.   Or when there are 
ultiple stages of a given flow (a -> b -> c), or when you have
two stages (async PostgreSQL and LDAP queries) which need to
be merged to generate a HTML page...

On Sun, Jul 13, 2003 at 12:24:34PM -0000, Moshe Zadka wrote:
| from __future__ import generators
| def os_path_walk(path, visit, arg):
|     d = defer.Deferred()
|     def walk(path):
|         files = os.listdir(path)
|         visit(arg, path, files)
|         for file in files:
|             yield None
|             if os.path.isdir(file):
|                 for _ in walk(os.path.join(path, file):
|                     pass
|     next = walk(path).next
|     def caller():
|         try:
|             next()
|         except StopIteration:
|             d.callback(None)
|         except:
|             d.errback(sys.exc_info()[1])
|         else:
|             reactor.callLater(0, caller)
|     caller()
|     return d

from __future__ import generators
from twisted.internet import reactor
from twisted.flow import flow
from twisted.python import util

def flowFactorial(n, steps = 10):
    coop = flow.Cooperate()
    acc = 1L
    while True:
        for i in range(steps):
            if 1 == n:
                yield acc
                return
            n, acc = n - 1, acc * n
        yield coop

d = flow.Deferred(flowFactorial(13))
d.addCallback(util.println)
d.addCallback(lambda _: reactor.stop())
reactor.run()