[Twisted-Python] Best strategies for pb Referenceables running long methods from callRemote

exarkun at twistedmatrix.com exarkun at twistedmatrix.com
Sat Mar 19 16:29:41 EDT 2011


On 06:48 pm, charlessolar at gmail.com wrote:
>Ah no, I use conch to connect to the remote machines.  My conch code 
>starts
>a remote python instance and runs some bootstrapping python code which
>connects to the PB Root.  Once connected I send the test code to the 
>remote
>process and start testing with PB.  When the PB remote method
>(remote_execute) is called from the users machine, that starts a new 
>thread
>with deferToThread to run the test code.  Since tests have multiple 
>parts
>that need to run simultaneously, some parts are delivered to the same
>machine, causing one machine to have to execute 5 or more parts 
>creating
>many twisted threads.
>The test code does its thing which stresses a network library.  It 
>creates
>servers and clients and makes sure everyone gets connected and gets all
>messages involved with the test.  It waits for all these events to 
>happen
>simultaneously so the twisted threads can be alive for very long 
>periods of
>time, and the test depends on the fact that all such threads are 
>running and
>not waiting in the thread pool overflow queue.
>I am planning on switching the multiple thread, one python design to
>multiple python, one thread, which I hope will allow all elements of 
>the
>test to run simultaneously easier than just simply increasing the 
>thread
>pool size.

Does creating servers and clients involve running child processes?  If 
so, you should be able to use reactor.spawnProcess instead of 
deferToThread and a blocking child process API.

But if it involves running arbitrary blocking Python code, then I don't 
see how you can avoid threads.  Switching to processes may help or it 
may hurt - it depends on the details of your workload (sometimes Python 
threads scale much worse than linearly making them worse than having an 
equal number of separate processes, but sometimes they're fine making 
them cheaper than processes; it all depends on what your code does).

Jean-Paul



More information about the Twisted-Python mailing list