[Twisted-Python] Re: Memory size of Deferreds

Martin Geisler mg at daimi.au.dk
Wed May 21 03:56:30 EDT 2008

Michał Pasternak <michal.dtz at gmail.com> writes:

>>   def add(x, y):
>>       sum = gatherResults([x, y])
>>       sum.addCallback(lambda results: results[0] + results[1])
>>       return sum
>> Multiplication is similar, but requires network communication --
>> that is why it returns a Deferred.
> I don't think that there's anything wrong with this example in terms
> of being written using Twisted Python.

Okay, thanks for looking at it!

>> The important point is that as much as possible is done in
>> parallel, that all operations run as soon as their operands are
>> available. Twisted made this extremely easy thanks to the
>> DeferredList and Deferred abstraction.
> This is the point where I think, that you answered your question. If
> you do stuff in parallel, you use more Deferreds and more memory.

Yes, I guess you are right... :-) The design tries to do everything at
once with as much parallism as possible. But since bandwidth is
limited, I guess I could achieve the same throughput by scheduling
things in a slower rate. Maybe using something like this:

  class map_parallel(Deferred):
      def __init__(self, iterable, func, max):
          self.iter = iterable
          self.func = func
          self.running = 0
          for _ in range(max):
              self.running += 1
      def run_next(self, arg):
              e = self.iter.next()
          except StopIteration:
              # The iterator is exhausted.
              self.running -= 1
              if self.running == 0 and not self.called:
                  # All callbacks have finished.
          # Pass the argument through unchanged.
          return arg

where map_parallel will map a function over a list of Deferreds, but
it will only run up to max function applications in parallel. I will
experiment with it, and if it works well, then it would be cool to
have a class like this in Twisted.

> I will repeat myself now: I would rather try to find other places to
> optimize VIFF, than reducing memory footprint of Deferred. I don't
> have a CS degree, I'm not a mathematician, but from what you wrote
> it seems that maybe it would be worth to try implementing something
> stack-based (like RPN), than the tree approach you seem to be doing
> now - of course if having a stack is something you can have using
> secure multiparty calculations.

Thanks for the ideas! You could certainly do what you suggest,
changing the schedule like that is okay from a security point of view.
And if I can delay allocating memory until needed by doing so, then it
will definitely be interesting to try.

Martin Geisler

VIFF (Virtual Ideal Functionality Framework) brings easy and efficient
SMPC (Secure Multi-Party Computation) to Python. See: http://viff.dk/.

More information about the Twisted-Python mailing list