[Twisted-Python] DeferredDict, defaultdict, map/reduce, silliness

[Terry Jones]

Hi Valentino

>>>>> "Valentino" == Valentino Volonghi <dialtone at gmail.com> writes:
Valentino> This is actually a very very nice pattern.

I thought it was too cute not to post, though I didn't really know what to
do with it. Something along the lines of map/reduce occurred to me quite
late. I bet there are other nice things you could do, mainly along the
lines of processing/bucketing a bunch of unpredictable input.

One thing I'd (probably) change would be to not use Deferred as the
argument to default dict, but to use a class that looked like it but with
the Deferred already called. That way you don't have the potentially very
long chain of callbacks building up. It's also partly a question of when
you want to do the work - each time you call addCallback, or at the end
when you've accumulated all the input and sent it off to its respective
callback chain.

So you could do something like this:

    def getDefault(init):
        def x():
            return defer.succeed(init)
        return x

and then the MapReduce class would look like

    class MapReduce(object):
        def __init__(self, init, cb):
            self.cb = cb
            self.d = defaultdict(getDefault(init))

Or, more flexibly, pass the defaultdict into the MapReduce class, giving
you the option of not having the callback chain already called, etc.

Anyway, I'm glad you like it, and hope you can do something nice with it.

Terry