[Twisted-Python] Memory size of Deferreds
mg at daimi.au.dk
Mon May 19 04:40:32 EDT 2008
I am working on a project called VIFF (http://viff.dk/) which does
secure multiparty computations. Twisted, and in particular the
Deferred class, has been a great tool for this!
Each variable in a computation is stored as a Deferred, and the
computation builds an expression tree based on these Deferreds, which
is then evaluated as the Deferreds call their callbacks.
What worries me is the size of a single Deferred. One user wanted to
do computations on very large lists of Deferreds and reported that his
programs used more memory than expected.
I tried to measure the size of a Deferred by creating large lists of
them and found that a single empty Deferred takes up about 200 bytes.
Adding a callback brings that up to about 500 bytes.
The discussion can be found here:
I might be measuring this the wrong way -- I simply looked at how the
memory size grew in 'top' when I created more and more lists.
As noted in the thread above, it helps defining the __slots__
attribute -- it saves the 144 bytes needed for the default object
dictionary. Maybe that could be done for Deferred?
Has anybody measured how much memory the C implementation users per
Deferred? I have not tried it out yet.
VIFF (Virtual Ideal Functionality Framework) brings easy and efficient
SMPC (Secure Multi-Party Computation) to Python. See: http://viff.dk/.
More information about the Twisted-Python