<html><body>On 11:35 pm, jarrod@vertigrated.com wrote:<br /><br />>There is a "backend" C module that our Twisted server front ends, and it is<br />>highly multi-threaded.<br />>So the T1000 is PERFECT for our application, except that now Twisted is the<br />>bottleneck. :-(<br /><br />This seems odd to me.<br /><br />If all the CPUs are going to be busy doing a multi-threaded back-end's work, and Twisted is just doing the I/O, then it seems the T1000 would still be a benefit.  The benchmark you mentioned was completely static; there was no backend library, no multithreaded CPU load.  Is the performance disparity similar when you're running actual workloads?<br /><br />Sure, Twisted isn't going to be able to dole out as much work as something optimized to balance the I/O management CPU across N+1 cores; but if those cores are going to be busy anyway in realistic use, then presumably having Twisted contending for all of them wouldn't be much of a performance boost.<br /><br />I actually do have a little experience with Twisted-*like* software on Solaris, although not Twisted itself.  The proprietary system which originally inspired Twisted's networking core was actually designed to run on Solaris, and took Sparc hardware advantages into account.  It still ran all of its I/O in a single thread.<br /><br />>So we either scrap our Twisted implementation and have to spend extra time<br />>on another network handling layer, or run 5 times as many instances of our<br />>server to service the same number of concurrent clients.<br /><br />Congratulations.  For years, I've been warning people that Twisted cannot transparently take full advantage of vertical scaling with SMP.  While I've heard a lot of uninformed whinging about how this is a huge problem, you are the first person to report an actual performance problem related to that fact :).<br /><br />Running 5 times as many instances of the server does make sense, and shouldn't have a significant downside.  The parallelism strategy I've used pretty much everywhere is multiprocessing rather than multithreading, and it works well.<br /><br />If the issue is that you don't want to have that many different open ports on each machine, would it be possible to have a small front-end server accept()ing and sending sockets to N+1 (where N is the number of cores) other Twisted processes?  I don't know how this might be accomplished on solaris, but if it's possible, it should be transparent to the clients and let Twisted itself take advantage of the hardware.  It would take some work, but not as much as a rewrite.<br /><br />Again, it seems weird to me that this is necessary if the back-end library is really utilizing all the CPUs already and you are not I/O bound.<br /></body></html>