[Twisted-Python] Looping in a deffered

exarkun at twistedmatrix.com exarkun at twistedmatrix.com
Sat Sep 5 10:17:00 EDT 2009


On 02:49 am, jjoske at nextdigital.com wrote:
>I'm trying to make a simple deferred that will continue get a message 
>of
>a message queue, process it and then wait for another message.
>
>While I can get the first message easily I am unable to work out how I
>can get a second message without creating a new deferred in the 
>printMsg
>function, which eventually gives me a maximum recursion depth exceeded.
>
>Basic code is a follows:
>
>from twisted.internet import reactor,defer
>
>def printMsg(msg):
>    print "Message is:"
>    print msg
>
>    deferred=getMsg()
>    deferred.addCallback(printMsg)
>
>def getMsg() :
>    d=defer.Deferred()
>    #replaced with code that actually goes to a queue to get the message
>    msg="This is a message"
>    d.callback(msg)
>    return d
>
>deferred=getMsg()
>deferred.addCallback(printMsg)
>reactor.run()
>
>Can any one point me on the right path to solve this?
>
>Thanks
>
>John

The above code is basically another way of writing this:

    def getMsg():
        return "This is a message"

    def printMsg(msg):
        print "Message is:"
        print msg
        printMsg(getMsg())

    printMsg(getMsg())

Hopefully it's obvious why this version of the code would fail 
eventually with a recursion error.  These are equivalent because 
d.addCallback(f) will call f immediately (ie, before addCallback 
returns) if d.callback(result) has already been called.

This only changes if getMsg (the original Deferred-returning one) ever 
returns a Deferred which has *not* already fired.  In such a case, the 
call stack will unwind back to the main loop and the code won't run any 
further until the Deferred returned by getMsg is called back.

I would expect the actual queue-retrieval version of the code to 
sometimes (or always) return a Deferred which has not yet been called 
back.  However, if you're seeing recursion errors, then this must not be 
the case.  Are you expecting the queue-retrieval version to return 
Deferreds that have already fired?

If you aren't, then I don't think I understand why you're encountering 
this error (or you're mistaken in your expectations).  An example closer 
to your actual program might help clear things up.

If you are, then you have two options.

First, you can stop using Deferreds and change the code from recursive 
to iterative.  eg,

    def getMsg():
        return "This is a message"

    def printMsg(msg):
        print "Message is:"
        print msg

    while True:
        printMsg(getMsg())

Of course, that's not very event loop friendly, so you might try making 
it cooperative by hooking it up to a scheduler:

    def messageIteration():
        printMsg(getMsg())
        reactor.callLater(0, messageIteration)

Or, factoring the specific scheduler policy out:

    def messagePrinter():
        while True:
            yield printMsg(getMsg())

    from twisted.internet.task import coiterate
    coiterate(messagePrinter())

Second, if there's a good reason to use Deferreds in this code (for 
example, there are multiple implementations of getMsg, some of which are 
synchronous, some of which are asynchronous, and preserving the 
interface between all of them is desirable), then you may want to use 
one of the above scheduler techniques as a trampoline so that you can 
avoid having the stack build up when a synchronous implementation is 
used:

    def messagePrinter():
        d = getMsg()
        d.addCallback(printMsg)
        d.addCallback(lambda ignored: reactor.callLater(0, 
messagePrinter))

Or you might use inlineCallbacks to define the trampoline:

    @inlineCallbacks
    def messagePrinter():
        while True:
            pringMsg(yield getMsg())

The idea here being that you never let recursion continue unbounded. 
You process one iteration and then let its call frames finish, return, 
and pop off the call stack so that they're out of the way for the next 
iteration.

Incidentally, there are a couple of other ways that Deferreds can trick 
you into hitting the call stack limit.  They can be addressed in ways 
similar to the ones I've described here.  However, they're also due to 
implementation limitations which hopefully will be removed someday.  The 
problem that you've run into here, though, doesn't seem to be due to an 
implementation limitation (at least not one that I can imagine any 
resolution to now) - it's just a pattern you need to avoid.

Hope this helps,

Jean-Paul



More information about the Twisted-Python mailing list