Thanks for the reply. I am using twisted conch to connect to the remote machines and start these tests, I am thinking that instead of using one connection for all testing I will open a new connection (and thus a new python) for each test. Like I said increasing the thread pool size worked well for fast machines but no so well on older ones. I am hoping that this behavior has more to do with the GIL and that 5 processes on a slow machine will operate better than 5 python threads. <br>
If not, then I will just have to write some sort of system to make sure I dont open too many threads on certain remote machines.<br><br>Thanks again<br><br><div class="gmail_quote">On Fri, Mar 18, 2011 at 8:40 PM, <span dir="ltr"><<a href="mailto:exarkun@twistedmatrix.com">exarkun@twistedmatrix.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div><div></div><div class="h5">On 10 Mar, 11:08 pm, <a href="mailto:charlessolar@gmail.com">charlessolar@gmail.com</a> wrote:<br>
>I am using PB to run remote methods in a testing system at my company.<br>
>The<br>
>code works very well but breaks down when I start running multiple<br>
>tests at<br>
>once. I have tracked this down to overflowing the thread pool on the<br>
>remote<br>
>machines. I am wondering if anyone might have better suggestions for<br>
>running long methods from a remote method.<br>
><br>
>I coded up a sample of what I am seeing here:<br>
><a href="http://pastebin.com/rBPp20Ms" target="_blank">http://pastebin.com/rBPp20Ms</a><br>
><br>
>Basically I have 1 server that calls remote_execute on many clients on<br>
>a<br>
>remote server. This remote_execute method starts a new method using<br>
>threads.deferToThread and returns the defer to make the server's<br>
>callRemote<br>
>defer wait until the remote long method end.<br>
>What I do in those methods is run test code that waits, blocks, sleeps,<br>
>and<br>
>all sorts of nasty things that make the thread take a while. In the<br>
>example<br>
>code I simply sleep for 20 seconds.<br>
><br>
>The problem I see with this code specifically is that I run out of<br>
>threads<br>
>on the pool and even though I wanted all execute methods to run at the<br>
>same<br>
>time, I see 10 run, then 10 more, then 10 more.. etc. The testing<br>
>depends<br>
>on all these methods being run at the same time as they run mechanisms<br>
>that<br>
>depend on each other and need everyone running. When I overflow the<br>
>thread<br>
>pool some methods do not run until other methods stop, which makes the<br>
>whole<br>
>test fail.<br>
<br>
</div></div>This is how the thread pool works. It has a maximum size, which limits<br>
the number of threads it will create to process work given to it.<br>
Beyond that number of concurrent tasks, things will begin to get queued<br>
up and wait for a free thread to execute them.<br>
<br>
Each task you give to the thread pool exclusively uses one of its<br>
threads for the entire duration of the task, regardless of what the task<br>
consists of.<br>
<div class="im">>I am not holding the GIL or blocking the reactor, which was the first<br>
>thing<br>
>I checked.<br>
><br>
>Setting reactor.suggestThreadPoolSize(50) does help, but I do not think<br>
>its<br>
>the best solution, and does not work very well on our slow and older<br>
>machines.<br>
<br>
</div>Using more threads is the only solution to the problem of not using<br>
enough threads. Alternatively, look for wards to process tasks without<br>
using threads.<br>
<br>
Jean-Paul<br>
<br>
_______________________________________________<br>
Twisted-Python mailing list<br>
<a href="mailto:Twisted-Python@twistedmatrix.com">Twisted-Python@twistedmatrix.com</a><br>
<a href="http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python" target="_blank">http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python</a><br>
</blockquote></div><br>