[Twisted-Python] Stackless Python Examples Revisited Re: InlineCallbacks

[Andrew Francis]

Hi Folks:

>Message: 4
>Date: Sat, 19 Apr 2008 11:47:38 -0700 (PDT)
>From: Andrew Francis <andrewfr_ice at yahoo.com>
>Subject: [Twisted-Python] Stackless Python Examples Re: InlineCallback
>    Friendly ?
>To: twisted-python at twistedmatrix.com
>Message-ID: <882470.36243.qm at web34201.mail.mud.yahoo.com>
>Content-Type: text/plain; charset=iso-8859-1

--- Andrew Francis <andrewfr_ice at yahoo.com> wrote:

>Based on Jean-Paul's explanation, here are some code
>snippets showing how to handle other protocols with
>Stackless Python and Twisted. This is a continuation
>of the technique I used in "Adventures" to prevent the
>reactor from deadlocking.

>Currently I am rethinking the techniques - I believe
>there is a way to reduce spawning tasklets - not that
>tasklets are all that expensive....


I have been experimenting with a new technique for tasklets to interact with Twisted protocol instances (the following is essentially what I was pestering the Twisted Team about at PyCon 2008. Sorry guys)

I have been exploring situations where a tasklet rendezvoused with Twisted (as opposed to a Twisted spawning a tasklet to fulfill a request). A lot of WS-BPEL constructs take this form....

In the old system, I ran the server protocol instance in its own tasklet. This was to prevent the reactor from blocking and preventing other tasklets from issuing Twisted calls. In turn, the tasklet and protocol instance/tasklet communicated via a channel. 

With this technique, I do not create an additional tasklet. Rather I pass back a deferred whose callback points to the reminder of the protocol instance's code. 

Although I have not finished testing, this technique seems to be about 20% faster. Again, implementing this is fairly straightforward. Here is a sketch.


class MyRequestHandler(http.Request):

    def __httpWrite__(self, status, message):
        self.setResponseCode(status)
        self.write(message.encode("utf-8"))
        self.finish()
        return


    def process(self):

        """
        create a deferred
        """
        myDeferred = defer.Deferred()
        myDeferred.addCallback(self.processReply)

        
        """
        we need a way for the protocol instance to tell tasklets that a
        connect has been made. Easiest way, just hand the protocol 
        instance a reference
        """
        status, message = \
        MyRequestHandler.someReferenceToProgramme(self.path, \
                                                  self.content.read(), \
                                                  myDeferred)
        if status != http.OK:
           print "ERROR!"
           self.__httpWrite__(status, message)
           myDeferred = None

        """
        if things went well, return a deferred 
        This is a Twisted way to circumvent blocking and waiting 
        for information
        """
        return myDeferred


    """
    the continuation 
    """
    def processReply(self, reply):
        self.__httpWrite__(200, reply[0])
        return


meanwhile, in the tasklet

def tasklet(...):
    .
    .
    .
    deferred = someFunction(somePath)
    # send a message 
    deferred.callback(message)
       
Although PyCon 2008 is over, I still plan to submit a paper that will better explain the various techniques in the "Adventures" talk. I am also writing simpler code to better illustrate the techniques and combine them with other tried-and-true techniques (i.e., Armstrong's blockOn). 

Cheers,
Andrew