[Twisted-Python] IPushProducer problem

[Gabriel Rossetti]

Jean-Paul Calderone wrote:
> On Tue, 13 Jan 2009 16:34:29 +0100, Gabriel Rossetti 
> <gabriel.rossetti at arimaz.com> wrote:
>> Hello everyone,
>>
>> I implemented a push-produce a while back and I though it works, but 
>> it didn't. When the msgs where spaced out, it worked, but if several 
>> msgs were sent one after the other, then things go bad (the msgs get 
>> mixed up)...
>>
>> I found this thread : 
>> http://www.twistedmatrix.com/pipermail/twisted-python 
>> /2007-March/014983.html
>>
>> and I think I ave the same problem, my code looks a bit like his :
>>
>>    def pauseProducing(self):
>>        """
>>        Pause the producer
>>        """
>>        self.__paused = True
>>
>>    def resumeProducing(self):
>>        """
>>        Resume the producer
>>        """
>>        self.__paused = False
>>
>>        while(not self.__paused and self.__startLimit < self.__total):
>>            data = self.__buf[self.__startLimit:self.__endLimit]
>>            self.transport.write(data)
>>            self.__startLimit = self.__endLimit
>>            self.__endLimit += self.__burstSize
>>            if(not self.__stream):
>>                break
>>
>>        if(not self.__paused or self.__startLimit >= self.__total):
>>            self.stopProducing()
>>
>>    def stopProducing(self):
>>        """
>>        Stop the producer
>>        """
>>        self.__paused = False
>>        self.__startLimit = 0
>>        self.__buf = None
>>        self.__total = None
>>        self.__endLimit = self.__burstSize
>>
>> and when I send a msg I use this :
>>
>> def sendMessage(self, msg):
>>        if domish.IElement.providedBy(msg):
>>            msg = msg.toXml()
>>              if isinstance(msg, unicode):
>>            msg = msg.encode('utf-8')
>>                      self.__buf = msg
>>        self.__total = len(msg)
>>        if(self.__stream):
>>            self.resumeProducing()
>>
>> I think what happens is that the code is re-entered, just like 
>> Jean-Paul suggested in the other thread. The thing is I didn't 
>> understand the solution, could somebody please explain it to me or 
>> suggest something else?
>
> It's extremely difficult to say without seeing a complete example and 
> without
> knowing exactly how the code is failing.  However, one thing I do see 
> is that
> it is not safe to call your `sendMessage` a second time before all of the
> data from the first call has been written.  Since you are using 
> `__buf´ to
> record the message to send, if you try to send another message, the first
> buffer will be clobbered.  Could this be the problem you're having?
>
> Perhaps consider appending to `__buf´ instead of overwriting it.  
> There are
> probably lots of other areas where this code could improve too, but not
> corrupting the internal state of the producer is a good start. :)
>
> Jean-Paul
>
Thanks for your answer Jean-Paul, yes, not currupting the internal state 
of my producer is a good start :-). I re-wrote it like this and it seems 
to work correctly now :

    def pauseProducing(self):
        """
        Pause the producer
        """
        self.__paused = True

    def resumeProducing(self):
        """
        Resume the producer
        """
        self.__paused = False

        while(not self.__paused and self.__buf):
            self.transport.write(self.__buf[:self.__burstSize])
            self.__buf = self.__buf[self.__burstSize:]
            if(not self.__stream):
                break

        if(not self.__paused or not self.__buf):
            self.stopProducing()

    def stopProducing(self):
        """
        Stop the producer
        """
        self.__paused = False

    def sendMessage(self, msg):
        if domish.IElement.providedBy(msg):
            msg = msg.toXml()
       
        if isinstance(msg, unicode):
            msg = msg.encode('utf-8')
               
        self.__buf += msg
        if(self.__stream and not self.__paused):
            self.resumeProducing()


Thanks again for pointing that out :-),
Gabriel