[Twisted-Python] Memory size of Deferreds

Michal Pasternak michal.dtz at gmail.com
Mon May 19 15:18:14 EDT 2008


2008/5/19 Martin Geisler <mg at daimi.au.dk>:
> What worries me is the size of a single Deferred. One user wanted to
> do computations on very large lists of Deferreds and reported that his
> programs used more memory than expected.
>
> I tried to measure the size of a Deferred by creating large lists of
> them and found that a single empty Deferred takes up about 200 bytes.
> Adding a callback brings that up to about 500 bytes.

As such thing as memory allocation is tightly tied to the runtime
environment, it is a bit hard to give reasonable answers without any
information about, at least, an OS and a CPU.

Did the memory in the case you mention grow constantly? Is that user
sure, that it was not a memory leak in his/hers code?

Deferreds, as you all already know, are a core concept that Twisted is
built upon. If you need to optimize such areas of Twisted, then, just
my suggestion - maybe you are using a wrong tool to do the job. If the
memory usage is a problem, then, well, maybe you are trying to do
things at once, when you should do them sequentially (eg. using a
generator). Maybe you should rather look into other areas of the
problem (eg. not try to optimize deferreds, but the code that uses
them). Maybe you should use something else than Deferreds for that
job. As for memory usage of such small objects like Deferreds, well...
Python is, well, Python. Python string or integer uses much more
memory, than C string or int. You can't tune Python string memory
usage, and you hardly can do anything about size of Deferred if that
becomes a problem. On the other hand, you should be able to write
optimal code with Python faster and memory footprint shouldn't be a
problem (from my experience, in many cases Twisted footprint is
suprisingly low).

In other words - if size of Deferred object is your only bottleneck,
then congratulations :-) I'd rather suggest optimizing other parts of
the code - as Deferreds are the core of Twisted, I suppose it could be
a bit hard to do anything about them, in the same way you can't do
anything about memory footprint of core Python objects.

Regards,
-- 
m




More information about the Twisted-Python mailing list