[Twisted-web] 100% CPU; what are my options?

Leif K-Brooks eurleif at gmail.com
Sat Jan 2 22:26:57 EST 2010


Hey,

I operate Omegle (http://omegle.com/), which is a COMET-driven
Twisted.Web app. (It doesn't use Nevow or anything; just Twisted.Web.)
It gets fairly heavy traffic; right now, it peaks daily at around 6500
concurrent users. Technical details:

- Hosted on a Linode 2880 virtual private server
(https://www.linode.com/); Ubuntu, 2880 MB of RAM, 4 cores available
- EPollReactor

Omegle currently runs at close to 100% CPU for most of the day, and
I'm a bit concerned about that. Here are some questions I have:

- How much more load might my app be able to take before it dies?
Currently, during peak load, a request from localhost is completed in
around 350ms; not ideal, but acceptable. I think anything under 1000ms
would be all right, but I'd prefer the latency to be as low as
possible, of course. Is this likely to scale linearly with traffic, or
what?
- I've looked at my code pretty closely, and it doesn't seem to be
doing anything particularly inefficient, but are there any tricks I
might be able to use to increase performance? Tweaking settings, etc.?
- Ultimately, I think my app will need to be able to use multiple
cores. If I have to, I'm going to either rewrite it in Erlang, or
rewrite it to load balance across multiple Twisted processes. Is there
anything easier I could do to allow the use of multiple cores?

Thanks a lot,
Leif



More information about the Twisted-web mailing list