[Twisted-Python] waitForDeferred Question

Jean-Paul Calderone exarkun at divmod.com
Sat Mar 11 14:44:21 EST 2006

On Sat, 11 Mar 2006 10:10:38 -0800, Brian Granger <bgranger at scu.edu> wrote:
> [snip]
>The problem is that the user will want to do:
>>>> result = computeSomething()
>>>> result = 2.0*computeOneThing()*math.sin(computeAnother())
>What should happen in these cases?  Ideally, the _entire_ command
>should become a "job" that evaluates in a "smart" manner.  That is,
>the final result shouldn't be computed until all other results are
>available.  Meanwhile, the prompt should retrn to the user so they can
>continue working.  But, I don't think this is possible w/o making
>changes to CPython itself.  The alternative is to simply make each
>command appear to block.

The difficulty you are encountering is that you want to use a language which is very much like Python as the input for your command shell, but which is not quite Python.  If you look at it this way, it should be clear that simply evaluating the input as Python will never give you a satisfactory solution.

One approach you might take, instead, is to parse the input using the compiler module, rewrite the AST so as to account for a Deferred at every step of the evaluation of the expression, and then compile the result into bytecode and execute that.

Taking your example:

    result = 2.0 * computeOneThing() * math.sin(computeAnother())

You would want to rewrite this to:

    _d1 = defer.maybeDeferred(computeAnother)
    _d1.addCallback(lambda result: math.sin(result))
    _d2 = defer.maybeDeferred(computeOneThing)
    _d2.addCallback(lambda result: result * 2.0)
    _d3 = defer.gatherResults([_d1, _d2])
    _d3.addCallback(lambda (left, right): left * right)
    _d3.addCallback(lambda result: assign("result", result))
    _d3.addCallback(lambda ign: continueREPL())

As you can see, this is a fairly simple mechanical transformation.  In addition to the steps I've taken above, you may also want to serialize the operations, in case allowing computeOneThing() and computeAnother() to run in parallel would be surprising to your users.


More information about the Twisted-Python mailing list