[Twisted-Python] Returning a Deferred as a result from another Deferred

[glyph at divmod.com]

On 06:34 am, rfugger at gmail.com wrote:
>On 10/5/07, David Reid <dreid at dreid.org> wrote:

>>deferToThread is best used for taking uncontrollably blocking APIs
>>(preferably C APIs that release GIL) and getting a Deferred back.  I
>>personally can not think of a single reason why you would ever take a
>>Deferred returning API and shuffle it off to a thread.

>See my original email for the specifics.  I receive a command on a
>connection, deferToThread for handling, which involves database
>access, and then callFromThread to send a message on another
>connection, which returns a Deferred that fires when the reply is
>received, triggering a reply to the command on the original
>connection.

I assume the briefest Python summary of your code looks like this:

def thingInThread():
    databaseResult = databaseWork()
    deferred = reactor.callFromThread(otherConnection.sendRequest,
                                      databaseResult)
    return deferred

def thingInReactor():
    return threads.deferToThread(thingInThread)

This isn't really idiomatic Twisted code, because while it is 
technically safe (thingInThread doesn't *do* anything with its Deferred) 
it is slightly misleading.  Deferreds are not thread safe, and there's 
nothing you *could* do with that Deferred in thingInThread; if you 
wanted to add a callback to it, for example, you can't.

What you actually want is something more like this:

def thingInReactor():
    deferred = threads.deferToThread(databaseWork)
    def tellOtherConnection(databaseResult):
        return otherConnection.sendRequest(databaseResult)
    deferred.addCallback(tellOtherConnection)
    return deferred

This deferred can safely have callbacks added to it and generally be 
treated normally, since it never sees a thread.  Also, now you don't 
have the problem where .callback() is being called with Deferred as an 
argument; instead, you are returning one Deferred to another, which 
results in the outer Deferred ("deferred") receiving the inner 
Deferred's result (whatever "otherConnection.sendRequest" would have 
fired with).