Geez .... what a HUGE duh moment. This will ultimately run on multiple remote servers, but, while doing initial testing, I have been running it on just my dev box. And, of course, I keep forgetting that twisted (well, python) doesn't make use of multiple cores. I wasted all yesterday trying to figure out why this was running the way it was.....<div>
<br></div><div>A million thanks.</div><div><br></div><div>Sean<br><br><div class="gmail_quote">On Fri, Jul 3, 2009 at 7:52 AM, Johann Borck <span dir="ltr"><<a href="mailto:johann.borck@densedata.com">johann.borck@densedata.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im">Sean Hollingsworth wrote:<br>
> Let me start off by warning everyone that I am extremely new to<br>
> twisted (I've been using it maybe a week) and it's highly likely some<br>
> things don't work the way I think they should. Or, I'm just using them<br>
> incorrectly.<br>
><br>
</div>> [..]<br>
<div class="im">><br>
> The problems:<br>
><br>
> Right now, for development and testing, I have three servers running<br>
> on localhost on three ports.<br>
</div>Are you running them in a single process?<br>
<div class="im">> When I run my test code, it seems that each Perspective Broker runs<br>
> sequentially, rather than concurrently. For example, if i watch the<br>
> debugging output from the server, I see the following:<br>
><br>
> server 0 processing dataval1<br>
> server 0 processing dataval2<br>
> server 0 processing dataval3<br>
> server 0 processing dataval4<br>
</div>> server 1 processing dataval*5*<br>
<div class="im">> server 1 processing dataval6<br>
> server 1 processing dataval7<br>
> server 2 processing dataval8<br>
> server 2 processing dataval9<br>
> server 2 processing dataval10<br>
><br>
> My understanding is that the perspective brokers would work<br>
> concurrently. Is that incorrect? My guess is that they should work<br>
> concurrently and I am just doing something wrong in my code, due to my<br>
> very, very limited understanding of how they work. Well, how much of<br>
> twisted works, really.<br>
><br>
<br>
> Below is my the relevant code for the client (I've taken out code that<br>
> just deals with prepping data or debugging). Please keep in mind that<br>
> this is mostly testing code, while I get a better understanding of how<br>
> perspective brokers work and was cobbled together from examples and<br>
> docs found online.<br>
><br>
<br>
</div>[...]<br>
<br>
With the client code, nothing seems wrong, since it doesn't enforce one<br>
job to be finished before starting the next one - the problem has to be<br>
on the server side. Actually I suspect you're running the same service<br>
on 3 ports but within one process, right?. If this is the case it's no<br>
wonder the order of jobs completed matches exactly the order they're<br>
arriving in because it all happens in a single thread. If you want<br>
parallel processing you will have to use multiple processes. As a side<br>
note (which doesn't seem to apply here): if the work they're doing is<br>
not CPU-bound, but IO-bound (meaning the work is nothing but waiting for<br>
something to complete, which for some reason can't be done in an<br>
asynchronous manner, the most prominent examples being blocking database<br>
interfaces), you also have the alternative of using threads.<br>
<br>
I think the basic problem is you're assuming twisted to somehow schedule<br>
the work to different cores/cpus, which it doesn't do by default. You<br>
might want to look at ampoule <a href="https://launchpad.net/ampoule" target="_blank">https://launchpad.net/ampoule</a>, which is a<br>
process pool, there was another similar thing called AsynQueue, but the<br>
site is down atm: <a href="http://foss.eepatents.com/AsynQueue" target="_blank">http://foss.eepatents.com/AsynQueue</a>.<br>
<br>
hth,<br>
Johann<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
Twisted-Python mailing list<br>
<a href="mailto:Twisted-Python@twistedmatrix.com">Twisted-Python@twistedmatrix.com</a><br>
<a href="http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python" target="_blank">http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python</a><br>
</blockquote></div><br></div>