[Twisted-Python] Best strategies for pb Referenceables running long methods from callRemote

exarkun at twistedmatrix.com exarkun at twistedmatrix.com
Fri Mar 18 21:40:38 EDT 2011


On 10 Mar, 11:08 pm, charlessolar at gmail.com wrote:
>I am using PB to run remote methods in a testing system at my company. 
>The
>code works very well but breaks down when I start running multiple 
>tests at
>once.  I have tracked this down to overflowing the thread pool on the 
>remote
>machines.  I am wondering if anyone might have better suggestions for
>running long methods from a remote method.
>
>I coded up a sample of what I am seeing here: 
>http://pastebin.com/rBPp20Ms
>
>Basically I have 1 server that calls remote_execute on many clients on 
>a
>remote server.  This remote_execute method starts a new method using
>threads.deferToThread and returns the defer to make the server's 
>callRemote
>defer wait until the remote long method end.
>What I do in those methods is run test code that waits, blocks, sleeps, 
>and
>all sorts of nasty things that make the thread take a while.  In the 
>example
>code I simply sleep for 20 seconds.
>
>The problem I see with this code specifically is that I run out of 
>threads
>on the pool and even though I wanted all execute methods to run at the 
>same
>time, I see 10 run, then 10 more, then 10 more.. etc.  The testing 
>depends
>on all these methods being run at the same time as they run mechanisms 
>that
>depend on each other and need everyone running.  When I overflow the 
>thread
>pool some methods do not run until other methods stop, which makes the 
>whole
>test fail.

This is how the thread pool works.  It has a maximum size, which limits 
the number of threads it will create to process work given to it. 
Beyond that number of concurrent tasks, things will begin to get queued 
up and wait for a free thread to execute them.

Each task you give to the thread pool exclusively uses one of its 
threads for the entire duration of the task, regardless of what the task 
consists of.
>I am not holding the GIL or blocking the reactor, which was the first 
>thing
>I checked.
>
>Setting reactor.suggestThreadPoolSize(50) does help, but I do not think 
>its
>the best solution, and does not work very well on our slow and older
>machines.

Using more threads is the only solution to the problem of not using 
enough threads.  Alternatively, look for wards to process tasks without 
using threads.

Jean-Paul



More information about the Twisted-Python mailing list