[Twisted-Python] __slots__ help (Re: Memory size of Deferreds)

Marcin Kasperski Marcin.Kasperski at softax.com.pl
Tue May 20 09:44:59 EDT 2008


It is not that difficult to see what is inside the deferred (spawn
some debugger and see):

(this may be a class variable)

 debug  (boolean)

(those surely are instance variables)

 callbacks (a list, initially empty, later containing tuples of tuples)
 called (int)
 paused (int)
 result (anything)
 timeoutCall (None or object)
 _runningCallbacks (boolean)
 _debugInfo (object of type DebugInfo)

Bare names of attributes take about 60 bytes (this is what could be
reclamied with __slots__), some of them are references, we have a list
(which inside contains complicated structures), ....

It could be an interesting experiment to define __slots__ for Deferred
class and see what is the memory impact.

Hmm, hmm. I did an experiment. Here is one version of the testing code:

    Cls = Deferred
    count = 250000
    import os
    pid = os.getpid()
    cmd = "ps -F %d" % pid
    a = [Cls() for _ in xrange(count)]
    os.system(cmd)
    b = [Cls() for _ in xrange(count)]
    os.system(cmd)
    c = [Cls() for _ in xrange(count)]
    os.system(cmd)

and here is one using __slots__:


    from twisted.internet.defer import timeout, Deferred

    class SlotsDeferred(object):
        __slots__ = ['debug', 'callbacks', 'called', 'paused', 'result', 'timeoutCall', '_runningCallbacks', '_debugInfo']

        # whole content of original Deferred class copied as-is here

    Cls = SlotsDeferred
    count = 250000
    import os
    pid = os.getpid()
    cmd = "ps -F %d" % pid
    a = [Cls() for _ in xrange(count)]
    os.system(cmd)
    b = [Cls() for _ in xrange(count)]
    os.system(cmd)
    c = [Cls() for _ in xrange(count)]
    os.system(cmd)

And here are the results:

(standard Deferred)

$ python testdef.py
UID        PID  PPID  C    SZ   RSS PSR STIME TTY      STAT   TIME CMD
marcink  23109 20536 92 15180 56992   1 15:40 pts/18   S+     0:00 python testdef.py
UID        PID  PPID  C    SZ   RSS PSR STIME TTY      STAT   TIME CMD
marcink  23109 20536 99 28374 109000  1 15:40 pts/18   S+     0:02 python testdef.py
UID        PID  PPID  C    SZ   RSS PSR STIME TTY      STAT   TIME CMD
marcink  23109 20536 99 41568 161020  1 15:40 pts/18   S+     0:05 python testdef.py

(Deferred with __slots__ added)

$ python testdef.py
UID        PID  PPID  C    SZ   RSS PSR STIME TTY      STAT   TIME CMD
marcink  23127 20536  0  7834 28068   1 15:41 pts/18   S+     0:00 python testdef.py
UID        PID  PPID  C    SZ   RSS PSR STIME TTY      STAT   TIME CMD
marcink  23127 20536 99 13683 51160   0 15:41 pts/18   S+     0:01 python testdef.py
UID        PID  PPID  C    SZ   RSS PSR STIME TTY      STAT   TIME CMD
marcink  23127 20536 99 19532 74256   1 15:41 pts/18   S+     0:02 python testdef.py

As one can see, the memory usage is ~ halved.

So ... maybe it would make sense to add __slots__ to deferreds, after all?

-- 
----------------------------------------------------------------------
| Marcin Kasperski   |  In any large change, 1/3 will think it is
| http://mekk.waw.pl | great, 1/3 will think it is stupid, and 1/3
|                    |               will wait (Reed)
----------------------------------------------------------------------





More information about the Twisted-Python mailing list