[Twisted-Python] Running callbacks in a non-main thread

Phil Mayers p.mayers at imperial.ac.uk
Fri Feb 3 19:31:19 EST 2006

Gabe Rudy wrote:
> Hey,
> I realize that the point of deferreds is to have process/network intensive 
> tasks work on in their own thread and send the results to callback function 

No. The exact opposite in fact. The purpose of a deferred is to act as a 
placeholder for a result that will be available later, and to call 
callbacks or errbacks that receive that result, in a single threaded 
event-driven system.

> in the main thread, but is there away to have a deferred process it's 
> callback in a non-main thread?

If I understand you, the deferred callback will take a long time. Your 
code is indicative of a misunderstanding of deferreds, which you need 
not feel bad about - they're not an easy concept at first.

Broadly, if you have code that looks like this under synchronous frameworks:

def operation():
   res = blockingCall()
   # the following function takes time
   res2 = doCalc(res)
   return res2.field

...that needs to be turned into:

def asyncOperation():
   deferred = asyncCall()
   deferred.addCallbacks(asyncOperation_2, log.err)
   return deferred

def asyncOperation_2(value):
   # doCalc is still expensive
   deferred = deferToThread(doCalc, value)
   deferred.addCallbacks(asyncOperation_3, log.err)
   # by returning a deferred, the deferred that calls us will be
   # paused until this "INNER" deferred finishes, so it all "just works"
   return deferred

def asyncOperation_3(doCalc_result):
   # this is a trivial reformatting operation
   return doCalc_result.field

More information about the Twisted-Python mailing list