Asynchronous Programming with Twisted

Deferreds

Twisted uses the Deferred object as a manager for your asynchronous callback sequence. Your client application attaches to the deferred object a reference to some function to which it has delegated the responsibility for dealing with the results of the asychronous request once those results are available. Your application should also entrust some function (possibly the same one) with the responsibility for dealing with an error that results from its request instead of data. Such an error-handling callback function is known as an errback.

Your application can also attach a series of functions that process the results and pass their own results on to the next guy. Such a series is known as a callback chain), and should be used with another a series of functions that are called if there is an error in the asychronous request (known as a series of errbacks or an errback chain). The asychronous library code calls the first callback when the result is available, or the first errback when an error occurs, and the Deferred object then hands the results of each callback or errback function to the next function in the chain.

The Problem that Deferreds Solve

It is the second class of concurrency problem — non-computationally intensive tasks that involve an appreciable delay — that Deferreds are designed to help solve. Functions that wait on hard drive access, database access, and network access all fall into this class, although the time delay varies.

Deferreds are designed to give Twisted programs a way to wait for data without hanging until that data arrives. They do this by providing a simple management interface libraries and applications to delegate the responsibility for dealing with possibly delayed data and errors to callbacks and errbacks, respectively. By returning a Deferred object, your Twisted-based library code knows that it can always make its results available by calling Deferred.callback. If an error crops up, your code will be able to have that unfortunate situation dealt with by calling Deferred.errback instead. Twisted-compatible applications have to do their part, however. They must set up the results handlers for the Deferred object by attaching to it the callbacks and errbacks they want called with results, in the order they want them called.

The basic idea behind Deferreds, and other asynchronous solutions as well, is to keep the CPU as active as possible. If one task is waiting on data, rather than have the CPU (and the program!) idle waiting for that data (a process normally called "blocking"), the program performs other operations in the meantime, confident that its callback and errback will deal with the data once it is ready to be processed. In Twisted, a function signals to the calling function that there is no immediate result by returning a Deferred as its result. When the data is available, the program activates the callbacks on that Deferred to process the data in sequence.

Deferreds - a signal that data is yet to come

In our email sending example above, a parent function calls a function to connect to the remote server. Asynchrony requires that this connection function return without waiting for the result so that the parent function can do other things. So how does the parent function or its controlling program know that the connection doesn't exist yet, and how does it use the connection once it does exist?

What Twisted uses to signal this situation is, of course, our versatile twisted.internet.defer.Deferred object. When the connection function returns, it signals that the operation is incomplete by returning a Deferred rather than the actual handle to the connection.

The Deferred has two purposes. The first is that it says "I am a signal that the result of whatever you wanted me to do is still pending." The second is that you can ask the Deferred to run things when the data does arrive.

You can picture a function that returns a Deferred as acting like a librarian who responds to a patron's question ("Are these mushrooms poisonous?") with a handwritten note saying, "I don't have your answer off the top of my head, but let me know where I can call you with the answer when I have it." The caller to the function does the equivalent of the patron scribbling a phone number on the note by attaching a callback to the Deferred. An equivalent of a deferred chain is where the patron writes several numbers on the note, and the person answering the phone at the first number responds to the answer ("They're highly poisonous") with another answer ("It's too late to use another recipe, our dinner party is canceled") that the librarian relays to whomever answers the phone (some disappointed dinner guest, perhaps) at the second number. Note that the library patron has been able to wander off and forget all about this matter in the meantime; such is the beauty of asynchronous programming!

Callbacks

The way you tell a Deferred what to do with the data once it arrives is by adding a callback — asking the Deferred to call a function once the data arrives.

One Twisted library function that returns a Deferred is twisted.web.client.getPage. In this example, we call getPage, which returns a Deferred, and we attach a callback to handle the contents of the page once the data is available:

from twisted.web.client import getPage

from twisted.internet import reactor

def printContents(contents):
    '''
    This is the 'callback' function, added to the Deferred and called by
    it when the promised data is available
    '''

    print "The Deferred has called printContents with the following contents:"
    print contents

    # Stop the Twisted event handling system -- this is usually handled
    # in higher level ways
    reactor.stop()

# call getPage, which returns immediately with a Deferred, promising to
# pass the page contents onto our callbacks when the contents are available
deferred = getPage('http://twistedmatrix.com/')

# add a callback to the deferred -- request that it run printContents when
# the page content has been downloaded
deferred.addCallback(printContents)

# Begin the Twisted event handling system to manage the process -- again this
# isn't the usual way to do this
reactor.run()

A very common use of Deferreds is to attach two callbacks. The result of the first callback is passed to the second callback:

from twisted.web.client import getPage

from twisted.internet import reactor

def lowerCaseContents(contents):
    '''
    This is a 'callback' function, added to the Deferred and called by
    it when the promised data is available. It converts all the data to
    lower case
    '''

    return contents.lower()

def printContents(contents):
    '''
    This a 'callback' function, added to the Deferred after lowerCaseContents
    and called by it with the results of lowerCaseContents
    '''

    print contents
    reactor.stop()

deferred = getPage('http://twistedmatrix.com/')

# add two callbacks to the deferred -- request that it run lowerCaseContents
# when the page content has been downloaded, and then run printContents with
# the result of lowerCaseContents
deferred.addCallback(lowerCaseContents)
deferred.addCallback(printContents)

reactor.run()

Error handling: errbacks

Just as an asynchronous function returns before its result is available, it may also return before it is possible to detect errors: failed connections, erroneous data, protocol errors, and so on. Just as you can add callbacks to a Deferred which it calls when the data you are expecting is available, you can add error handlers ('errbacks') to a Deferred for it to call when an error occurs and it cannot obtain the data:

from twisted.web.client import getPage

from twisted.internet import reactor

def errorHandler(error):
    '''
    This is an 'errback' function, added to the Deferred which will call
    it in the event of an error
    '''

    # this isn't a very effective handling of the error, we just print it out:
    print "An error has occurred: <%s>" % str(error)
    # and then we stop the entire process:
    reactor.stop()

def printContents(contents):
    '''
    This a 'callback' function, added to the Deferred and called by it with
    the page content
    '''

    print contents
    reactor.stop()

# We request a page which doesn't exist in order to demonstrate the
# error chain
deferred = getPage('http://twistedmatrix.com/does-not-exist')

# add the callback to the Deferred to handle the page content
deferred.addCallback(printContents)

# add the errback to the Deferred to handle any errors
deferred.addErrback(errorHandler)

reactor.run()

Conclusion

In this document, you have:

1. seen why non-trivial network programs need to have some form of concurrency;
2. learnt that the Twisted framework supports concurrency in the form of asynchronous calls;
3. learnt that the Twisted framework has Deferred objects that manage callback chains;
4. seen how the getPage function returns a Deferred object;
5. attached callbacks and errbacks to that Deferred; and
6. seen the Deferred's callback chain and errback chain fire.

See also

Since the Deferred abstraction is such a core part of programming with Twisted, there are several other detailed guides to it: