[Twisted-Python] Transparent pooling of deferreds to be fired upon another deferred firing

Terry Jones terry at jon.es
Wed Apr 22 20:22:27 EDT 2009

I have a need to make sure that a deferred-returning function is not called
multiple times, but I want to be able to write code without having to worry
about that.

To be more concrete:

Suppose I have a function called alertSupervisor that takes a string
message for the supervisor and which returns a deferred. I want to be able
to call that function from anywhere in my code, and I always want to get a
deferred back, but if there's already a call in progress to tell the
supervisor the same thing then I don't want to bug him/her by sending
another message, but I do want to know when the original call fires.

So code fragment A calls

  d = alertSupervisor("We have a problem in the boiler room!")

and before the deferred has fired, code fragment B notices the same
underlying problem, and also calls

  d = alertSupervisor("We have a problem in the boiler room!")

I want A and B's deferreds to both be called back with the result/failure.
If I can write it like the above it has the advantage of looking like
regular code and no-one has to think about what's going on. Underneath
there's a pool of deferreds whose firing is triggered by the firing of the
single original deferred.

Note that I specifically only want to pool deferreds like this if there is
a call already underway (i.e., the original deferred has not fired). That
implies you could not pool calls to functions that return via defer.succeed
or defer.fail since those deferreds have already fired.  If someone calls
alertSupervisor("We have a problem in the boiler room!") and their deferred
fires, then a subsequent call to alertSupervisor("We have a problem in the
boiler room!") should get through to the supervisor, as per normal.

This may all seem a bit esoteric, but I have several uses for it. In
particular when the service providing alertSupervisor lives on one machine
and is being called via RPC by A and B, both on separate machines.  As for
how to nicely make RPC calls like this, see Thrift [1] which now has
built-in support for Twisted.

Here's a first cut at a solution:

    from twisted.internet import defer
    from twisted.python import failure

    class DeferredPooler(object):
        def __init__(self, func):
            self.pool = {}
            self.func = func

        def _callOthers(self, result, key):
            if isinstance(result, failure.Failure):
                for d in self.pool[key]:
                for d in self.pool[key]:
            del self.pool[key]
            return result

        def __call__(self, *args, **kwargs):
            key = (args, hash(tuple(sorted(kwargs.items()))))
            if key in self.pool:
                d = defer.Deferred()
                return d
                self.pool[key] = []
                d = self.func(*args, **kwargs)
                assert isinstance(d, defer.Deferred)
                d.addBoth(self._callOthers, key)
                return d

You use the class transparently, via a decorator:

    from twisted.internet import reactor

    def x(*args, **kwargs):
        print 'x called: args = %r, kwargs = %r' % (args, kwargs)
        d = defer.Deferred()
        reactor.callLater(2, d.callback, args)
        return d

    def printer(what):
        print 'printer received:', what

    def stop(_):

    if __name__ == '__main__':

        d1 = x('fred')

        d2 = x('fred')

        d = defer.DeferredList([d1, d2])

If you run this you'll see x is called only once, but d1 and d2 are each
fired properly.

For more Twisted pleasure/pain, you can also try out funky things like:

        d1 = x('fred')

        d2 = x('fred')

Here there are 4 calls to x, but only 2 get through.

I could say more, as usual. But I'll wait to see if there's any reaction.
One issue to consider further is the simplistic use of hash.  


[1] Thrift: http://incubator.apache.org/thrift/

More information about the Twisted-Python mailing list