[Twisted-Python] What is the best way to implement "need more data" semantics?

甜瓜 littlesweetmelon at gmail.com
Thu Jun 14 05:03:32 EDT 2007

My goal is writing a specific protocol. The data transfered between
client and server are mixed with text and binary. I override the
method dataReceived. When it get some new data, it will call a certain
parseFunc according to the current state of the protocol, such as:
Step 1: dataReceived will call parseGreetingMessage(), and then call
Step 2: dataReceived will call parseHandShaking(), and then call
Step 3: dataReceived will call parseSecurity(), and then call ReplySecurity()

However, in some cases, the data received from the server is not
completely fetched by the client, so the data passed to parseFunc will
be treated as incomplete, and parseFunc will try to return and wait
for more data. Currently, my code for doing this seems very ulgy ^_^.
Could you please give me some suggestions to make the implementation
of the "need more data" semantics more clearly and more easily?

The problem is that the message boundary in my protocol is not as
simple as LineReceiver. Each step, I should use different parse to
determine whether I get the whole data. For example:
parseErrorString: The data compose an leading 4byte integer size plus
following string characters. If the length of collected data is
shorter than (4 + size of string), it will be treated as incomplete.
parseBitmap: The data compose a bitmap header and following bitmap
data. If the length of collected data is shorter than (sizeof(header)
+ headerwidth*header.height*bits_per_pixel), it will be treated as

My method is:
parseFunc parses the data while testing the completeness. If
incomplete occrred, parseFunct return. When new data arrived, the
dataReceived method will call parseFunc again, and "re-parse" the
previous data as well as new data. However, sometimes, re-parse is
time consuming. I want to find a way to hold the result for
already-parsed data. Maybe it will be similar to "generator":

# In class MyProtocol(protocol.Protocol):  pseudo-code
def parseFunc(...):
    # parse header ...
    if not completeness:
        yield   # need more data, go into data waiting state.
    # If new data arrived, dataReceived will be called, and then
    # call parseFunc again to continue parsing.
    # parse following data...
    if not completeness:
        yield   # need more data

def dataReceived(s):
    self.data += s


More information about the Twisted-Python mailing list