[Twisted-Python] Best strategies for pb Referenceables running long methods from callRemote
charlessolar at gmail.com
Fri Mar 18 22:06:11 EDT 2011
Thanks for the reply. I am using twisted conch to connect to the remote
machines and start these tests, I am thinking that instead of using one
connection for all testing I will open a new connection (and thus a new
python) for each test. Like I said increasing the thread pool size worked
well for fast machines but no so well on older ones. I am hoping that this
behavior has more to do with the GIL and that 5 processes on a slow machine
will operate better than 5 python threads.
If not, then I will just have to write some sort of system to make sure I
dont open too many threads on certain remote machines.
On Fri, Mar 18, 2011 at 8:40 PM, <exarkun at twistedmatrix.com> wrote:
> On 10 Mar, 11:08 pm, charlessolar at gmail.com wrote:
> >I am using PB to run remote methods in a testing system at my company.
> >code works very well but breaks down when I start running multiple
> >tests at
> >once. I have tracked this down to overflowing the thread pool on the
> >machines. I am wondering if anyone might have better suggestions for
> >running long methods from a remote method.
> >I coded up a sample of what I am seeing here:
> >Basically I have 1 server that calls remote_execute on many clients on
> >remote server. This remote_execute method starts a new method using
> >threads.deferToThread and returns the defer to make the server's
> >defer wait until the remote long method end.
> >What I do in those methods is run test code that waits, blocks, sleeps,
> >all sorts of nasty things that make the thread take a while. In the
> >code I simply sleep for 20 seconds.
> >The problem I see with this code specifically is that I run out of
> >on the pool and even though I wanted all execute methods to run at the
> >time, I see 10 run, then 10 more, then 10 more.. etc. The testing
> >on all these methods being run at the same time as they run mechanisms
> >depend on each other and need everyone running. When I overflow the
> >pool some methods do not run until other methods stop, which makes the
> >test fail.
> This is how the thread pool works. It has a maximum size, which limits
> the number of threads it will create to process work given to it.
> Beyond that number of concurrent tasks, things will begin to get queued
> up and wait for a free thread to execute them.
> Each task you give to the thread pool exclusively uses one of its
> threads for the entire duration of the task, regardless of what the task
> consists of.
> >I am not holding the GIL or blocking the reactor, which was the first
> >I checked.
> >Setting reactor.suggestThreadPoolSize(50) does help, but I do not think
> >the best solution, and does not work very well on our slow and older
> Using more threads is the only solution to the problem of not using
> enough threads. Alternatively, look for wards to process tasks without
> using threads.
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Twisted-Python