[Twisted-Python] dataReveived() buffer best practice?

[John Santos]

I'm doing something very similar.


I have (greatly simplified, because I'm doing lots of other stuff 
with the data):

    def applicationDataReceived(self, data):

	global savedText, delayCounter
        newline = "\n"
        text = savedText + data
        savedText = ""
        # normalize text (replace escapes, search for and normalize newlines)
        text = text.replace("\000","")
        text = text.replace("\033","$")
        text = text.replace("\r\n",newline)
        text = text.replace("\r",newline)
        lines = text.split(newline)
        for line in lines[:-1]:
            if len(line):
                write( line+newline )
        savedText = lines[-1]
        # (if savedText <> "" then we are waiting for more...)

        def printPartialMessage(myCounter):
            global savedText, savedTimestamp, delayCounter
            #print "delay routine called %i %i" % (myCounter, delayCounter)
            if myCounter == delayCounter:
                if savedText <> "":
                    newline = "\n"
                    write( savedText+" (terminator not seen)"+newline )
                    savedText = ""
                delayCounter = 0
            # If the passed counter (myCounter) equals the current value of
            # the delay counter, then we know the savedText is that left over
            # when we were queued.  Otherwise, more text has come in and
            # caused the counter to be incremented and more calls to be
            # made to printPartialMessage.  We'll let them handle it.

        if savedText <> "":
            delayCounter += 1
            reactor.callLater(60, printPartialMessage, delayCounter)



I'm actually writing this to a time-stamped log file, but I've removed
all the code to maintain the time stamps, create and roll over the log
files, etc.

If you are willing to wait indefinitely for partial lines, you can get rid
of the "printPartialMessage()" deferred, and the conditional call to it. 

delayCounter is a "pending calls to printPartialMessage()" counter (reset
to 0 when there are none), because the only one we are interested in is
the last one.  Instead, the deferred could be canceled when new text
arrives, but I think the ability to cancel defereds is newer than this
code.  I just let the deferreds exexute and dismiss them immediately if
they aren't current.

The constant "60" in the reactor.callLater() is the number of seconds I'm
willing to wait for a partial message to receive more data.

You might not want to normalize the text (since I'm printing to a text log
file, I didn't want embedded escape sequences to cause conniptions when
the log file was printed to a terminal), or you might want to use a 
different line terminator. 

Also, for conch.telnet, I think you want to override 
"applicationDataReceived()" and not "dataReceived()" because 
"dataReceived()" sees all the telnet protocol negotiations, which you
might want to let conch.telnet handle.



-- 
John Santos
Evans Griffiths & Hart, Inc.
781-861-0670 ext 539