[Twisted-Python] Another approach to allowing __init__ to work with Deferreds

[Terry Jones]

I posted to this list back in Nov 2008 with subject:
A Python metaclass for Twisted allowing __init__ to return a Deferred

Briefly, I was trying to find a nice way to allow the __init__ method of a
class to work with deferreds in such a way that methods of the class could
use work done by __init__ safe in the knowledge that the deferreds had
completed.  E.g., if you have

    class X(object):
        def __init__(self, host, port):
            def final(connection):
                self.db = connection
            d = makeDBConnection(host, port)
            d.addCallback(final)

        def query(self, q):
            return self.db.runQuery(q)

Then when you make an X and call query on it, there's a chance the deferred
wont have fired, and you'll get an error.  This is just a very simple
illustrative example.  There are many more, and this is a general problem
of the synchronous world (in which __init__ is supposed to prepare a
fully-fledged class instance and cannot return a deferred) meeting the
asynchronous world in which we would like to (and must) use deferreds.

The earlier thread:

  http://twistedmatrix.com/pipermail/twisted-python/2008-November/018600.html

Although I learned a lot in that thread, I wasn't completely happy with any
of the solutions. Some of the things that still bugged me are in posts
towards the end of the thread:

  http://twistedmatrix.com/pipermail/twisted-python/2008-November/018624.html
  http://twistedmatrix.com/pipermail/twisted-python/2008-November/018634.html

The various approaches we took back then all boiled down to waiting for a
deferred to fire before the class instance was fully ready to use. When
that happened, you had your instance and could call its methods.

I had also thought about an alternate approach: having __init__ add a
callback to the deferreds it dealt with to set a flag in self and then have
all dependent methods check that flag to see if the class instance was
ready for use. But that 1) is ugly (too much extra code); 2) means the
caller has to be prepared to deal with errors due to the class instance not
being ready, and 3) adds a check to every method call. It would look
something like this:

    class X(object):
        def __init__(self, host, port):
            self.ready = False
            def final(connection):
                self.db = connection
                self.ready = True
            d = makeDBConnection(host, port)
            d.addCallback(final)

        def query(self, q):
            if not self.ready:
                raise IAmNotReadyException()
            return self.db.runQuery(q)

That was too ugly for my taste, for all of the above reasons, most
especially for forcing the unfortunate caller of my code to handle
IAmNotReadyException.


Anyway.... fast forward 6 months and I've hit the same problem again. It's
with existing code, in which I would like an __init__ to call something
that (now, due to changes elsewhere) returns a deferred. So I started
thinking again, and came up with a much cleaner way to do the alternate
approach via a class mixin:

    from twisted.internet import defer

    class deferredInitMixin(object):
        def wrap(self, d, *wrappedMethods):
            self.waiting = []
            self.stored = {}

            def restore(_):
                for method in self.stored:
                    setattr(self, method, self.stored[method])
                for d in self.waiting:
                    d.callback(None)

            def makeWrapper(method):
                def wrapper(*args, **kw):
                    d = defer.Deferred()
                    d.addCallback(lambda _: self.stored[method](*args, **kw))
                    self.waiting.append(d)
                    return d
                return wrapper

            for method in wrappedMethods:
                self.stored[method] = getattr(self, method)
                setattr(self, method, makeWrapper(method))

            d.addCallback(restore)


You use it as in the class Test below:

    from twisted.internet import defer, reactor

    def fire(d, value):
        print "I finally fired, with value", value
        d.callback(value)

    def late(value):
        d = defer.Deferred()
        reactor.callLater(1, fire, d, value)
        return d

    def called(result, what):
        print 'final callback of %s, result = %s' % (what, result)

    def stop(_):
        reactor.stop()


    class Test(deferredInitMixin):
        def __init__(self):
            d = late('Test')
            deferredInitMixin.wrap(self, d, 'f1', 'f2')

        def f1(self, arg):
            print "f1 called with", arg
            return late(arg)

        def f2(self, arg):
            print "f2 called with", arg
            return late(arg)


    if __name__ == '__main__':
        t = Test()
        d1 = t.f1(44)
        d1.addCallback(called, 'f1')
        d2 = t.f1(33)
        d2.addCallback(called, 'f1')
        d3 = t.f2(11)
        d3.addCallback(called, 'f2')
        d = defer.DeferredList([d1, d2, d3])
        d.addBoth(stop)
        reactor.run()


Effectively, the __init__ of my Test class asks deferredInitMixin to wrap
some of its methods. deferredInitMixin stores the original methods away and
replaces each of them with a function that immediately returns a deferred.
So after __init__ finishes, code that calls the now-wrapped methods of the
class instance before the deferred has fired will get a deferred back as
usual (but see * below). As far as they know, everything is normal.  Behind
the scenes, deferredInitMixin has arranged for these deferreds to fire only
after the deferred passed from __init__ has fired.  Once that happens,
deferredInitMixin also restores the original functions to the instance. As
a result there is no overhead later to check a flag to see if the instance
is ready to use. If the deferred from __init__ happens to fire before any
of the instance's methods are called, it will simply restore the original
methods.  Finally (obviously?) you only pass the method names to
deferredInitMixin that depend on the deferred in __init__ being done.

BTW, calling the methods passed to deferredInitMixin "wrapped" isn't really
accurate. They're just temporarily replaced.


I quite like this approach.  It's a second example of something I did in
http://twistedmatrix.com/pipermail/twisted-python/2009-April/019522.html in
which a pool of deferreds is accumulated and they're all fired when another
deferred fires. It's nice because you don't reply with an error and there's
no need for locking or other form of coordination - the work you need done
is already in progress, so you get back a fresh deferred and everything
goes swimmingly.

* Minor note: the methods you wrap should probably be ones that already
return deferreds. That way you always get a deferred back from them,
whether they're temporarily wrapped or not. The above mixin works just fine
if you ask it to wrap non-deferred-returning functions, but you have to
deal with the possibility that they will return a deferred (i.e., if you
call them while they're wrapped).

Comments welcome / wanted.

Terry