[Twisted-Python] Deffered question

[Andrew Bennetts]

On Tue, Oct 21, 2003 at 02:32:32PM +0300, soso wrote:
> Hi all, 
> 
> I need to implement a command based protocol, that is a client sends a command
> to the server and waits for the response. Since the response computation might
> be time consuming I have to choose between 2 solutions:
> - use Twisted deffereds
> - or write my own implementation, a TCP based threaded server. I'm not worried
> about scaling since it will not be used by more than 10 clients at a time.
> 
> And the question is, how are deffereds actually implemented? My guess is that
> there are some worker threads, besides the main thread. Or?

Deferreds aren't magical at all; the implementation doesn't involve threads,
or anything behind the scenes at all.  By themselves, they don't magically
turn synchronous (i.e. blocking) code into asynchronous (i.e. non-blocking)
code.

They're simply a very convenient abstraction around the concept of a
"deferred result" that eases management of callbacks that operate on that
result.

See also:
    http://twistedmatrix.com/pipermail/twisted-python/2003-June/004333.html

> I actually started implementing my own server, Twisted seems too huge. But I may
> change my mind, I'm new to both Python and Twisted.

Twisted is huge, but then so is the Python standard library.  Twisted is
great for writing servers in.  The upcoming 1.1 release contains a tutorial
that should hopefully make understanding what you need to do to write a
server easier; you can preview the CVS version of it at:
    http://twistedmatrix.com/users/warner/doc-latest/howto/tutorial.html
(That's currently an 800kB page!)

Can you elaborate more on what the "response computation" involves?  If you
can, we can probably sketch out some code that does what you need.

Here's an example off the top of my head, though, that will hopefully help
you a little (warning, barely tested code):

----
from twisted.protocols import basic
from twisted.python import log
from twisted.internet import reactor, threads
from twisted.internet.protocol import ServerFactory

def square(n):
    return n*n

class SquaringProtocol(basic.LineReceiver):
    """A protocol that receives a number, squares it and sends back the
    result.
    
    This drops the connections after one request-response cycle,
    HTTP/1.0-style. 
    """
    
    def lineReceived(self, line):
        try:
            number = int(line)
        except ValueError:
            self.sendLine("%r is not a number" % line)
            return
        
        deferred = threads.deferToThread(square, number)
        # Convert the result to a string
        deferred.addCallback(str)
        # Send it
        deferred.addCallback(self.sendLine)
        # Write any errors to the log file
        deferred.addErrback(log.err)
        # Drop the connection
        deferred.addBoth(lambda x: self.transport.loseConnection())


class RepeatingSquaringProtocol(basic.LineReceiver):
    """A protocol that receives numbers, squares them, and sends back the
    results.
    
    This allows multiple requests to be made on a single connection, without
    waiting for responses.  The requests will queue up on the server, and be
    returned (in the order they were requested) as they are processed.  It
    is up to the client to disconnect.
    """
    
    def connectionMade(self):
        self.busy = 0
        self.queue = []
        
    def lineReceived(self, line):
        try:
            number = int(line)
        except ValueError:
            self.sendLine("%r is not a number" % line)
            return
        
        self.queueRequest(number)
    
    def queueRequest(self, number):
        if self.busy:
            self.queue.append(number)
        else:
            self.processRequest(number)

    def processRequest(self, number):
        deferred = threads.deferToThread(square, number)
        deferred.addCallback(str)
        deferred.addCallback(self.requestDone)
        deferred.addErrback(log.err)

    def requestDone(self, result):
        self.sendLine(result)
        if self.queue:
            number = self.queue.pop(0)
            self.processRequest(number)
        else:
            self.busy = 0
        

if __name__ == '__main__':
    from twisted.python import log
    import sys
    log.startLogging(sys.stdout)
    factory = ServerFactory()
    factory.protocol = SquaringProtocol
    reactor.listenTCP(9001, factory)
    factory = ServerFactory()
    factory.protocol = RepeatingSquaringProtocol
    reactor.listenTCP(9002, factory)
    reactor.run()

----

This example has two slightly different versions of a simple protocol that
squares numbers.  I'm pretending that squaring a number is a slow
calculation that is best run in a worker thread -- imagine your computation
where I'm using 'square'.

Hopefully it gives you an idea of how easy this sort of thing is in Twisted.

-Andrew.