[Twisted-Python] Learning Twisted

[Jason Diamond]

Sorry for the length of this post but I learn best by trying to
explain what I'm learning (even if nobody's listening). Twisted seems
very cool but also *huge* and unlike any framework I've used before so
I thought I'd post my first experiences with it in the hopes that I
could be corrected where needed (and maybe even help people other new
like me).

Here's the deal: I want to write an IMAP4 client and Twisted's
IMAP4Client [1] looks much more featureful than Python's imaplib.

Since I'm writing a client, I started out by reading the "Writing a
TCP client" HOWTO [2] where I learned, by looking at the ircLogBot
example, that I needed to define a subclass of IMAP4Client and a
subclass of ClientFactory. This is what I came up with:

from twisted.internet import reactor, protocol
from twisted.protocols import imap4

class MyIMAP4Client(imap4.IMAP4Client):

    def connectionMade(self):
        imap4.IMAP4Client.connectionMade(self)
        print "connectionMade"

class MyIMAP4ClientFactory(protocol.ClientFactory):

    protocol = MyIMAP4Client

f = MyIMAP4ClientFactory()
reactor.connectTCP("server", 143, f)
reactor.run()

Running this works (I see "connectionMade" printed to the console).

IMAP4Client has a login method so I modified my connectionMade method
to look like this:

    def connectionMade(self):
        imap4.IMAP4Client.connectionMade(self)
        print "connectionMade"
        self.login("user", "password")

But I couldn't tell if it worked or not. That's when I read that login
returns a "deferred whose callback is invoked if login is
successful". So I read the "Using Deferreds" HOWTO [3] and learned
that I needed to pass in a function to addCallback on the deferred
returned by calling login. So connectionMade now looks like this:

    def connectionMade(self):
        imap4.IMAP4Client.connectionMade(self)
        print "connectionMade"
        d = self.login("user", "password")
        d.addCallback(loginCallback)

loginCallback looks like this:

def loginCallback(d):
    print "loginCallback:", d

Note that this is *not* a method of the MyIMAP4Client class. But it
worked! I got this printed to the console:

connectionMade
loginCallback: ([], 'OK LOGIN Ok.')

I don't know, however, what the tuple represents. What's that empty
list? In a perfect world, would this be explained in the IMAP4Client
documentation? (I'm assuming that every callback would be different.)

I thought it sucked that my loginCallback function wasn't a member of
my class but I thought I'd see what error I'd get if I made it one so
I changed the two methods to look like this:

    def loginCallback(self, d):
        print "loginCallback:", d

    def connectionMade(self):
        imap4.IMAP4Client.connectionMade(self)
        print "connectionMade"
        d = self.login("user", "password")
        d.addCallback(self.loginCallback)

And that worked, too! At this point, I don't understand *why* this
works but it's awfully nice that it does. ("Using Deferreds" shows one
example of a callback method but doesn't discuss it in any way.)

Next, I wanted to try actually doing something useful. So I called
self.select from loginCallback so that I could see how many messages
exist in my INBOX.

    def loginCallback(self, d):
        print "loginCallback:", d
        de = self.select("INBOX")
        de.addCallback(self.selectCallback)

    def selectCallback(self, d):
        print "selectCallback:", d

This is the output I got for selectCallback:

selectCallback: {'EXISTS': 323, 'PERMANENTFLAGS': ('$MDNSent', 
'NonJunk', '\\*',
 '\\Draft', '\\Answered', '\\Flagged', '\\Deleted', '\\Seen'), 
'READ-WRITE': 1,
'FLAGS': ('$MDNSent', 'NonJunk', '\\Draft', '\\Answered', '\\Flagged', 
'\\Delete
d', '\\Seen', '\\Recent'), 'UIDVALIDITY': 1076206465, 'RECENT': 0}

Apparently, the parameter to the select callback is a dict. The
documentation for select actually mentions some of the keys. It looks
like I could use the EXISTS key to get how many messages exist in my
INBOX:

    def selectCallback(self, d):
        print "I have %d messages in my INBOX." % d["EXISTS"]

Now all I have to do is logout (up until now, I've been terminating
the script with Ctrl-C):

    def selectCallback(self, d):
        print "I have %d messages in my INBOX." % d["EXISTS"]
        de = self.logout()
        de.addCallback(self.logoutCallback)

    def logoutCallback(self, d):
        sys.exit(0)

This isn't correct, however, since the call to sys.exit prints an
exception to the console and, more distressingly, does *not* exit the
script.

At this point I realize that I'm in some sort of event loop (started
by calling reactor.run) and need to tell this loop to stop. So I look
at the "Reactor basics" HOWTO [4] and find my way to IReactorCore [5]
which documents a stop method:

    def logoutCallback(self, d):
        reactor.stop()

This works wonderfully although I have no idea if the connection to
the server is actually being gracefully closed. So I override
connectionLost to find out:

    def connectionLost(self, reason):
        imap4.IMAP4Client.connectionLost(self)
        print "connectionLost"

Now "connectionLost" is the last thing I see on my console before the
script exits. Nice.

I also override both sendLine and lineReceived so that I can see the
conversation with the IMAP server to see if I'm actually logging out
before closing the connection:

    def sendLine(self, line):
        imap4.IMAP4Client.sendLine(self, line)
        print line

    def lineReceived(self, line):
        imap4.IMAP4Client.lineReceived(self, line)
        print line

Perfect.

Just for reference, here's the program I ended up with:

from twisted.internet import reactor, protocol
from twisted.protocols import imap4

debug = 0

class MyIMAP4Client(imap4.IMAP4Client):

    def connectionMade(self):
        imap4.IMAP4Client.connectionMade(self)
        if debug: print "connectionMade"
        d = self.login("user", "password")
        d.addCallback(self.loginCallback)

    def loginCallback(self, d):
        de = self.select("INBOX")
        de.addCallback(self.selectCallback)

    def selectCallback(self, d):
        print "I have %d messages in my INBOX." % d["EXISTS"]
        de = self.logout()
        de.addCallback(self.logoutCallback)

    def logoutCallback(self, d):
        reactor.stop()

    def connectionLost(self, reason):
        imap4.IMAP4Client.connectionLost(self)
        if debug: print "connectionLost"

    def sendLine(self, line):
        imap4.IMAP4Client.sendLine(self, line)
        if debug: print line

    def lineReceived(self, line):
        imap4.IMAP4Client.lineReceived(self, line)
        if debug: print line

class MyIMAP4ClientFactory(protocol.ClientFactory):

    protocol = MyIMAP4Client

f = MyIMAP4ClientFactory()
reactor.connectTCP("server", 143, f)
reactor.run()

So how'd I do for a Twisted newbie? Am I on the path to enlightenment?
Are there any obvious errors in what I've interpreted so far? Is this
a "good" implementation for this feature or is their a more
Twisted-ish approach I need to strive for?

Am I correct in assuming that the IMAP4Client documentation needs some
buffing up? (Maybe I can help with that.) I didn't look at any source
code (other than in the HOWTOs) in implementing this so I think that's
a good sign. But as I mentioned above, I'm not clear on how I know
what the signature for my callbacks should be or what the parameters
to those callbacks mean in some cases.

Also, it's not obvious to me how protocol implementors decide when to
have users override a method versus having them use callbacks. I know
I'm working on client code but knowing this might help me know how to
use their code. Is it a matter of personal preference? The ircLogBot
example didn't have any callbacks using deferreds but it looks like I
had no choice when using IMAP4Client.

Thanks for reading this far!

-- Jason

[1]
http://twistedmatrix.com/documents/current/api/twisted.protocols.imap4.IMAP4Client.html

[2]
http://twistedmatrix.com/documents/current/howto/clients

[3]
http://twistedmatrix.com/documents/current/howto/defer.html

[4]
http://twistedmatrix.com/documents/current/howto/reactor-basics.html

[5]
http://twistedmatrix.com/documents/TwistedDocs/TwistedDocs-1.2.0/api/twisted.internet.interfaces.IReactorCore.html