Ah no, I use conch to connect to the remote machines. My conch code starts a remote python instance and runs some bootstrapping python code which connects to the PB Root. Once connected I send the test code to the remote process and start testing with PB. When the PB remote method (remote_execute) is called from the users machine, that starts a new thread with deferToThread to run the test code. Since tests have multiple parts that need to run simultaneously, some parts are delivered to the same machine, causing one machine to have to execute 5 or more parts creating many twisted threads. <br>
The test code does its thing which stresses a network library. It creates servers and clients and makes sure everyone gets connected and gets all messages involved with the test. It waits for all these events to happen simultaneously so the twisted threads can be alive for very long periods of time, and the test depends on the fact that all such threads are running and not waiting in the thread pool overflow queue.<br>
I am planning on switching the multiple thread, one python design to multiple python, one thread, which I hope will allow all elements of the test to run simultaneously easier than just simply increasing the thread pool size. <br>
<br>Charles<br><br><div class="gmail_quote">On Sat, Mar 19, 2011 at 10:42 AM, <span dir="ltr"><<a href="mailto:exarkun@twistedmatrix.com">exarkun@twistedmatrix.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
On 02:06 am, <a href="mailto:charlessolar@gmail.com">charlessolar@gmail.com</a> wrote:<br>
>Thanks for the reply. I am using twisted conch to connect to the<br>
>remote<br>
>machines and start these tests, I am thinking that instead of using one<br>
>connection for all testing I will open a new connection (and thus a new<br>
>python) for each test.<br>
<br>
I'm not sure I follow. In your earlier message you were talking about<br>
PB and deferToThread. How does this fit with your Conch usage? You're<br>
not using Conch in the function passed to deferToThread, are you?<br>
<br>
Jean-Paul<br>
>Like I said increasing the thread pool size worked<br>
>well for fast machines but no so well on older ones. I am hoping that<br>
>this<br>
>behavior has more to do with the GIL and that 5 processes on a slow<br>
>machine<br>
>will operate better than 5 python threads.<br>
>If not, then I will just have to write some sort of system to make sure<br>
>I<br>
>dont open too many threads on certain remote machines.<br>
><br>
>Thanks again<br>
<br>
_______________________________________________<br>
Twisted-Python mailing list<br>
<a href="mailto:Twisted-Python@twistedmatrix.com">Twisted-Python@twistedmatrix.com</a><br>
<a href="http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python" target="_blank">http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python</a><br>
</blockquote></div><br>