[Twisted-Python] Re: Using deferreds when writing across unreliable network

[Martin Geisler]

"John Aherne" <johnaherne at rocs.co.uk> writes:

Hi John

I have no concrete solution to what you describe, but I can try and
explain how Deferreds work.

> So should my callback or errback fire without me having to initiate
> any action other than setting up the deferred. And should I be using
> a deferred at all for this situation.

First, a Deferred is an amazingly powerful concept, but the
implementation is surpricingly simple. I once posted a simple
explaination for my own project:

  http://article.gmane.org/gmane.comp.cryptography.viff.devel/26

The basic idea is that a Deferred is a container for event handling
functions, called callbacks in Twisted. Python has function pointers,
so it is easy to make such a class:

  class MiniDeferred:
      def __init__(self):
          self.callbacks = []
          self.value = None

      def addCallback(self, func, *extra_args):
          self.callbacks.append((func, extra_args))

      def callback(self, initial_value):
          self.value = initial_value
          for func, extra_args in self.callbacks:
              self.value = func(self.value, *extra_args)
          return self.value

      def __str__(self):
          if self.value is None:
              return "MiniDeferred (no value yet)"
          else:
              return "MiniDeferred (value: %s)" % (self.value,)

This class skips the error handling (errbacks), but I believe it
illustrates the core ideas of a Deferred. You can use it like this:

  >>> x = MiniDeferred()
  >>> print x
  MiniDeferred (no value yet)
  >>> x.addCallback(lambda value: value**3)
  >>> x.callback(-3)
  >>> print x
  MiniDeferred (value: -27)

The crucial thing is that "someone" or "something" must call the
callback method on your Deferred. The general rule to follow is that
it is the code that creates a Deferrred which has the reponsibility of
calling the callback method at the correct time.

So when you use the getPage function you get a Deferred back. Behind
the scenes the getPage function has setup things in such a way that
the callback method of the returned Deferred is called when the page
has arrived -- you must not invole callback yourself in this case.

But if you create a Deferred yourself, then it is up to you to make a
call to the callback method when ready.

And example could be this mini implementation of the DeferredList
class:

  class MiniDeferredList(MiniDeferred):
      def __init__(self, deferreds):
          MiniDeferred.__init__(self)
          self.results = [None] * len(deferreds)
          self.missing_results = len(deferreds)
          for index, deferred in enumerate(deferreds):
              deferred.addCallback(self._callback_fired, index)

      def _callback_fired(self, result, index):
           self.results[index] = result
           self.missing_results -= 1
           if self.missing_results == 0:
               self.callback(self.results)
           return result

Here the _callback_fired method invokes the callback method when all
deferreds have fired.

I hope this helps a bit -- it helped me to look at the implementation
in defer.py and see that it was less scary than I had imagined :-)


-- 
Martin Geisler

VIFF (Virtual Ideal Functionality Framework) brings easy and efficient
SMPC (Secure Multiparty Computation) to Python. See: http://viff.dk/.