[Twisted-Python] Reducing memory footprint of twisted app?

[Andrew Bennetts]

On Thu, Aug 17, 2006 at 09:37:39PM -0400, jarrod roberson wrote:
> one of the things we did, and saw an approximately 30% REDUCTION in
> memory foot print was add ing __slots__ definations to all he objects
> we were creating in graphs.
> 
> This isn't twisted specific, so it should apply to any python application.
> 
> Granted we have hundreds of thousands of objects in the graph. But it
> did make a noticiable change in the foot print.

Right, __slots__ can be helpful.

Some more advice that isn't Twisted specific:

It's very helpful to understand which objects are taking up the memory.  If you
know that, not can have a good idea if __slots__  will actually help before you
clutter your code with them, but you can perhaps realise that you shouldn't even
have 100000 simultaneous Request objects when you only have 1000 connections at
the time -- in my experience helping people on IRC, it's quite common that
there's accidentally a reference being kept to every request object (or
similar), thus causing memory leaks despite Python's garbage collection.  Saving
30% of memory on 100000 objects isn't anywhere near as good as saving 99% of
those objects from being needed in the first place!

If you understand what the culprits are, you can also decide that not only are
__slots__ helpful, you can also analyse those objects to figure out if they are
keeping more state than they really need.

And in fact, you can try speculatively adding __slots__ to a type of object as
an indirect way to see if a particular type is a major contributor to your
memory use or not -- if adding __slots__ to Foo doesn't help, there probably
aren't a significant number of instances contributing to the memory use.

Basically, I really strongly think people should *understand* their performance
issues so they can fix them better, rather than just blindly doing the
equivalent of "gcc -O9" and considering it solved.  It depends on the available
time and requirements, of course; if a quick band-aid is all that's needed, then
fair enough.  But I find it usually pays off to throughly understand what you're
fixing.

That said, if you need hundreds of thousands of objects in memory, __slots__ is
one of the simplest ways to improve memory consumption I know of :)

-Andrew.