[Twisted-Python] debugging a memory leak

[exarkun at twistedmatrix.com]

On 01:24 am, johann.borck at densedata.com wrote:
>Alec Matusis wrote:
>>In desperation of not finding the real memory leak on the production 
>>server,
>>
>>I wrote a test server that I can push to arbitrary high RSS memory. I 
>>am far
>>from sure if this the same leak that I observe in production, but I 
>>would
>>like to understand what this one is.
>>This is the server code:
>>
>>Server.py:
>>
>>import twisted.protocols.basic
>>from twisted.internet.protocol import Factory
>>from twisted.internet import reactor
>>
>>class HiRate(twisted.protocols.basic.LineOnlyReceiver):
>>         MAX_LENGTH = 20000
>>
>>         def lineReceived(self, line):
>>                 if line == 'get':
>>                         out = 'a'*4000+'\r\r'
>>                         self.transport.write(out)
>>
>>
>>
>>factory = Factory()
>>factory.protocol = HiRate
>>
>>reactor.listenTCP(8007, factory, backlog=50, interface='10.18.0.2')
>>reactor.run()
>>[...]
>>To reproduce the memory leak, I either need two machines with fast LAN
>>between them (since the client program takes 100% CPU), or possibly 
>>one
>>machine with a dual core CPU (I have not tried that). It is important 
>>that
>>client.py is given a separate CPU to run.
>>When the length of the response from the server is sufficient,  (out =
>>'a'*4000+'\r\r' ,  4000 is enough in my case), the RSS of the server 
>>process
>>starts to leak without a bound.
>>If you introduce a small delay in the client (#time.sleep(.001)), the 
>>leak
>>does not occur.
>Hi Alec,
>I wouldn't call that a memory leak. In your example  ( SC/s = (n* CS) /
>s ;  n > 1, SC: bytes sent to the client, CS: bytes sent to the server 
>)
>there is some CS that satifies the condition SC/s > [available
>bandwidth]. For all CS that exceed that limit, the data will be first
>queued in the Send-Q of the socket and then inside twisted, because the
>Send-Q is full. That's the reason why you see increasing memory usage,
>but it's not a leak, it is a server that can not cope with this kind of
>DOS attack. Twisted can not handle this situation for you because that
>would mean it had to decide which packets, connections, etc. to drop.
>Such decisions have to be made by your application.

You can get notifications about when your send buffers are full by 
registering a producer with the transport. 
http://twistedmatrix.com/documents/current/core/howto/producers.html 
describes the APIs involved.  This will help you make these decisions in 
your application.

Jean-Paul