[Twisted-Python] Memory size of Deferreds

Martin Geisler mg at daimi.au.dk
Mon May 19 04:40:32 EDT 2008


Hello,

I am working on a project called VIFF (http://viff.dk/) which does
secure multiparty computations. Twisted, and in particular the
Deferred class, has been a great tool for this!

Each variable in a computation is stored as a Deferred, and the
computation builds an expression tree based on these Deferreds, which
is then evaluated as the Deferreds call their callbacks.

What worries me is the size of a single Deferred. One user wanted to
do computations on very large lists of Deferreds and reported that his
programs used more memory than expected.

I tried to measure the size of a Deferred by creating large lists of
them and found that a single empty Deferred takes up about 200 bytes.
Adding a callback brings that up to about 500 bytes.

The discussion can be found here:

  http://thread.gmane.org/gmane.comp.cryptography.viff.devel/232

I might be measuring this the wrong way -- I simply looked at how the
memory size grew in 'top' when I created more and more lists.

As noted in the thread above, it helps defining the __slots__
attribute -- it saves the 144 bytes needed for the default object
dictionary. Maybe that could be done for Deferred?

Has anybody measured how much memory the C implementation users per
Deferred? I have not tried it out yet.

  http://twistedmatrix.com/trac/ticket/2245

-- 
Martin Geisler

VIFF (Virtual Ideal Functionality Framework) brings easy and efficient
SMPC (Secure Multi-Party Computation) to Python. See: http://viff.dk/.





More information about the Twisted-Python mailing list