[Twisted-web] load balancing and performance

Andrea Arcangeli andrea at cpushare.com
Sat Jan 29 07:55:50 MST 2005


I successfully load balanced the static server with pythondirector.

However it has a few problems:

1) it cannot handle ssl
2) even if it can handle ssl I still need to share the session cookies
   and I'm thinking that using sql or atop to do that isn't necessary:
   the session doesn't need to survive a reboot, a reboot would be a visible
   disruption of the services anyway, so I'm thinking to write a little
   session daemon that only stores the cookies using pb to do that.
   In theory shared memory would be much more efficient in smp, but at least
   this socket model has the advantage of scaling in the cluster too
3) the load balancer misses an API with the real webserver to pass up the
   client IP address, that's annoying, especially the logs gets screwed

Other than that the speed of the load balancer is excellent, and I get
exactly the double number of pages rendered per second (after fixing a
small bug in pythondirector that confused ab2).

Perhaps I'm going with wrong priorities though, the major offender is
compy, compy must be dropped from Nevow ASAP :). Leaving it as a
compatibility API may be ok, but internally compy can't invoke
getComponents anymore if we do care about writing remotely optimal code.
We should try with the new zope.interfaces API first and see if it
underperforms so horribly as getInterfaces does.

Secondly I'm looking into caching the html and to render some fragment only
once every 10 seconds in the background (so the downloads will never
have to wait for a rendering of some mostly static fragment anymore).

So overall there's an huge room for improvements. What do other people
think?



More information about the Twisted-web mailing list