[Twisted-Python] A Python metaclass for Twisted allowing __init__ to return a Deferred

glyph at divmod.com glyph at divmod.com
Mon Nov 3 07:49:12 EST 2008

On 2 Nov, 11:48 pm, terry at jon.es wrote:
>Very briefly, I wrote a metaclass to allow you to write classes whose
>__init__ method uses deferreds. Your __init__ can create deferreds, 
>functions that return deferreds, and of course return a deferred itself
>(that's the whole point). Your class instance wont be available until 
>the deferred your __init__ returns has fired.

As a general stylistic thing, I've been writing a lot more classmethods 
lately to determine arguments to __init__, rather than trying to make 
__init__ itself do interesting tricks.  I can't find a name for this 
"design pattern", so let me describe it:

One very common use-case is that we have some object - let's say an RFC 
5322 email address - which is typically created from a string.  An 
idiomatic way to do that might be like this:

    import rfc822

    class Address:
        def __init__(self, addrstr):
            l = list(rfc822.AddressList(addrstr))
            if len(l) != 1:
                raise ValueError("Too many or too few addresses.")
                desc, addr = l[0]
                self.description = desc
                self.localpart, self.domain = addr.split("@")

But this is problematic.  With this class, it's hard to convert from a 
different format of storing email addresses that has already been 
parsed.  In order to create an Address from, i.e., a database record 
containing a description, localpart, and domain, I now need to smash 
everything back into a string, worrying about trivia like quoting; or I 
need to resort to hacks like calling __new__ instead of __init__.  It 
makes testing more difficult: in my tests I need to start having 
formatted email addresses in strings instead of simply creating Address 
objects.  If this class were hypothetically a bit smarter and dealt 
nicely with unicode, my tests would need to learn about email-address 
quoting rules in order to generate addresses with non-ASCII characters, 
rather than leaving that logic entirely in the Address class.  Ugly all 

However, I can pull the parsing logic out and separate it from the 
initialization logic, and all of that gets much easier:

    class Address:
        def __init__(self, localpart, domain, description):
            self.localpart = localpart
            self.domain = domain
            self.description = description

        def fromString(cls, addrstr):
            l = list(rfc822.AddressList(addrstr))
            if len(l) != 1:
                raise ValueError("Too many or too few addresses.")
                desc, addr = l[0]
                loc, dom = addr.split("@")
                return cls(loc, dom, desc)

With this improved class, I can easily create Address objects in other 
ways from other code.  Since it's a classmethod rather than a function, 
it's just as friendly to inheritance as a constructor; perhaps even 
moreso.  It opens the door to the evolution of other creation methods, 
fromXXX classmethods, without breaking the constructor's signature or 
changing the fromString method.

You don't give a concrete example in your blog post, but I can imagine 
that all these points apply twice over to any code that would use 
Deferreds.  An __init__ that returns a Deferred means that in the 
testing case, not only is there no way to directly construct the object 
you want, there might be no way to even get one without spinning the 
reactor.  What is that Deferred doing?  Maybe there's no way to get one 
without actually generating network traffic!  Obviously, not an ideal 
scenario.  For the tests for the code making the deferred request 
itself, there will obviously need to be fake sources of data, but for 
other tests that just want to interact with one of your objects, direct 
construction is pretty much always easier.

However, thanks for sharing nonetheless.  Although I wouldn't use it 
personally, your code makes an interesting rhetorical point.  There's a 
great deal of whinging that goes on around Deferreds being hard to work 
with.  This metaclass is just another in a long line of tools that says 
"see?  it really isn't so hard to deal with a Deferred if you need to."

More information about the Twisted-Python mailing list