[Twisted-Python] Strange behavior when transferring large strings

Gabriel Rossetti mailing_lists at evotex.ch
Wed Apr 23 04:41:29 EDT 2008

Gabriel Rossetti wrote:
> Gabriel Rossetti wrote:
>> Hello,
>> I have been trying to use the producer/consumer paradigm to transfer 
>> large strings. I've been having trouble because I have strange things 
>> going on, but I now suspect it isn't the P/C that's not working, 
>> because it works fine with smaller strings than the ones I'm having 
>> trouble with, so I suspect the problem comes from elsewhere. What 
>> happens is it starts off fine, but then the destination at some point 
>> only gets part of the string but the source has already sent 
>> everything, closed it's transport and called stopProducing(). Could 
>> there be some internal buffering problem, like I'm sending too much 
>> data at too high of a throughput, or maybe something like some sort 
>> of max time a socket can stay open or something? If I tell it to 
>> split the string up into smaller parts, it doesn't change anything, 
>> if I make the string smaller, whatever the size of each chunk, it 
>> transfers fine.
>> Thanks,
>> Gabriel
> I removed the P/C code and used the previous version and the problem 
> persists, if I send a XML msg (very small XML part) with a data 
> payload containing a string generated like this :
>    initialMsg = "".join([str(x) for x in range(5000)])
> which is 10374 bytes of data (including the XML), this makes me 
> suspect that there is maybe some sort of buffer or something that is 
> full and doesn't have the time to empty and twisted stops working or 
> something. I'm using the 2.5 version since Ubuntu hasn't upgraded (and 
> apparently won't upgrade until the lenny) their version yet and I 
> can't get it to compile from source (yes, I installed python-dev and 
> build-essentials), so maybe it's a problem only to this version. Does 
> anyone have an Idea of what the problem is? I'm using the following 
> code to send messages, I don't think that's the problem but you never 
> know :
>    def sendMessage(address, port, message, needAnswer=False):
>        d = defer.Deferred() if needAnswer else None
>              class MessageSender(Protocol):
>            def sendMessage(self, msg):
>                if domish.IElement.providedBy(msg):
>                    msg = msg.toXml()
>                       if isinstance(msg, unicode):
>                    msg = msg.encode('utf-8')
>                                  self.transport.write(msg)
>                          def dataReceived(self, data):
>                d.callback(data)
>                  def gotProtocol(proto):
>            proto.sendMessage(message)
>            if(not needAnswer):
>                proto.transport.loseConnection()
>        c = ClientCreator(reactor, MessageSender)
>        c.connectTCP(address, port).addCallback(gotProtocol)
>        return d
>   Gabriel
Ok, so apparently it (Twisted, Python, the OS?) can buffer and send up 
to 16k, after that it splits the message up, thus this would explain the 
lockup, since the app expects a whole XML message and can't process the 
second part correctly. So if I understand it correctly, even when I use 
the P/C paradigm, it waits until the buffer is full before sending the 
message, so when I get the message on the other side, I only get part of 
it, the other half is a new message/data arrival, and thus like I said 
above, the XML is not correct.

Does anyone have an idea of how to solve this?


More information about the Twisted-Python mailing list