[Twisted-Python] Best strategies for pb Referenceables running long methods from callRemote

[Charles Solar]

Thanks for the reply.  I am using twisted conch to connect to the remote
machines and start these tests, I am thinking that instead of using one
connection for all testing I will open a new connection (and thus a new
python) for each test.  Like I said increasing the thread pool size worked
well for fast machines but no so well on older ones.  I am hoping that this
behavior has more to do with the GIL and that 5 processes on a slow machine
will operate better than 5 python threads.
If not, then I will just have to write some sort of system to make sure I
dont open too many threads on certain remote machines.

Thanks again

On Fri, Mar 18, 2011 at 8:40 PM, <exarkun at twistedmatrix.com> wrote:

> On 10 Mar, 11:08 pm, charlessolar at gmail.com wrote:
> >I am using PB to run remote methods in a testing system at my company.
> >The
> >code works very well but breaks down when I start running multiple
> >tests at
> >once.  I have tracked this down to overflowing the thread pool on the
> >remote
> >machines.  I am wondering if anyone might have better suggestions for
> >running long methods from a remote method.
> >
> >I coded up a sample of what I am seeing here:
> >http://pastebin.com/rBPp20Ms
> >
> >Basically I have 1 server that calls remote_execute on many clients on
> >a
> >remote server.  This remote_execute method starts a new method using
> >threads.deferToThread and returns the defer to make the server's
> >callRemote
> >defer wait until the remote long method end.
> >What I do in those methods is run test code that waits, blocks, sleeps,
> >and
> >all sorts of nasty things that make the thread take a while.  In the
> >example
> >code I simply sleep for 20 seconds.
> >
> >The problem I see with this code specifically is that I run out of
> >threads
> >on the pool and even though I wanted all execute methods to run at the
> >same
> >time, I see 10 run, then 10 more, then 10 more.. etc.  The testing
> >depends
> >on all these methods being run at the same time as they run mechanisms
> >that
> >depend on each other and need everyone running.  When I overflow the
> >thread
> >pool some methods do not run until other methods stop, which makes the
> >whole
> >test fail.
>
> This is how the thread pool works.  It has a maximum size, which limits
> the number of threads it will create to process work given to it.
> Beyond that number of concurrent tasks, things will begin to get queued
> up and wait for a free thread to execute them.
>
> Each task you give to the thread pool exclusively uses one of its
> threads for the entire duration of the task, regardless of what the task
> consists of.
> >I am not holding the GIL or blocking the reactor, which was the first
> >thing
> >I checked.
> >
> >Setting reactor.suggestThreadPoolSize(50) does help, but I do not think
> >its
> >the best solution, and does not work very well on our slow and older
> >machines.
>
> Using more threads is the only solution to the problem of not using
> enough threads.  Alternatively, look for wards to process tasks without
> using threads.
>
> Jean-Paul
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20110318/1518c09b/attachment.htm