[Twisted-Python] Best strategies for pb Referenceables running long methods from callRemote

[Charles Solar]

Ah no, I use conch to connect to the remote machines.  My conch code starts
a remote python instance and runs some bootstrapping python code which
connects to the PB Root.  Once connected I send the test code to the remote
process and start testing with PB.  When the PB remote method
(remote_execute) is called from the users machine, that starts a new thread
with deferToThread to run the test code.  Since tests have multiple parts
that need to run simultaneously, some parts are delivered to the same
machine, causing one machine to have to execute 5 or more parts creating
many twisted threads.
The test code does its thing which stresses a network library.  It creates
servers and clients and makes sure everyone gets connected and gets all
messages involved with the test.  It waits for all these events to happen
simultaneously so the twisted threads can be alive for very long periods of
time, and the test depends on the fact that all such threads are running and
not waiting in the thread pool overflow queue.
I am planning on switching the multiple thread, one python design to
multiple python, one thread, which I hope will allow all elements of the
test to run simultaneously easier than just simply increasing the thread
pool size.

Charles

On Sat, Mar 19, 2011 at 10:42 AM, <exarkun at twistedmatrix.com> wrote:

> On 02:06 am, charlessolar at gmail.com wrote:
> >Thanks for the reply.  I am using twisted conch to connect to the
> >remote
> >machines and start these tests, I am thinking that instead of using one
> >connection for all testing I will open a new connection (and thus a new
> >python) for each test.
>
> I'm not sure I follow.  In your earlier message you were talking about
> PB and deferToThread.  How does this fit with your Conch usage?  You're
> not using Conch in the function passed to deferToThread, are you?
>
> Jean-Paul
> >Like I said increasing the thread pool size worked
> >well for fast machines but no so well on older ones.  I am hoping that
> >this
> >behavior has more to do with the GIL and that 5 processes on a slow
> >machine
> >will operate better than 5 python threads.
> >If not, then I will just have to write some sort of system to make sure
> >I
> >dont open too many threads on certain remote machines.
> >
> >Thanks again
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20110319/818940d8/attachment.htm