[Twisted-Python] deferToThread and reactor loop

exarkun at twistedmatrix.com exarkun at twistedmatrix.com
Thu Nov 1 18:03:31 EDT 2012


On 29 Oct, 05:58 pm, tobias.oberstein at tavendo.de wrote:
>Hi,
>
>I've got a question regarding
>
>deferToThread / runWithConnection
>
>I have a network server that accepts RPCs and forwards those to a 
>relational database calling stored procedures.
>
>The call of the stored procedures is happening via "runWithConnection" 
>.. that is on a background thread pool (since that DBs driver is 
>blocking).
>
>Now, when there is a large number of RPCs coming in roughly at once, 
>the latency of each call forwarded to the RDBMs increased into the 10's 
>of ms.
>
>I have created a self-contained test program without DB etc that I 
>think reproduces the "issue":
>
>https://github.com/oberstet/scratchbox/blob/master/python/twisted/threadpoollatency/test.py
>
>**************
>Am I correct that "deferToThread" does not immediately forward the call 
>to a background thread, but only the next time the reactor loop runs?
>**************

Hi Tobias,

Sorry, I haven't had a chance to read your minimal example in any 
detail.  However, I can direct you to the implementation of 
deferToThread:

http://twistedmatrix.com/trac/browser/trunk/twisted/python/threadpool.py#L119

Notice the `self.q.put(o)`.  This matches up with the call to 
`self.q.get` in the same module:

http://twistedmatrix.com/trac/browser/trunk/twisted/python/threadpool.py#L158

Together, these bits of source should demonstrate that there's no 
waiting for a reactor iteration before the work is enqueued.  The work 
goes into the Queue instance, and instantly any worker thread is free to 
grab it.

Jean-Paul



More information about the Twisted-Python mailing list