[Twisted-Python] debugging a memory leak

[Johann Borck]

Alec Matusis wrote:
> In desperation of not finding the real memory leak on the production server,
>
> I wrote a test server that I can push to arbitrary high RSS memory. I am far
> from sure if this the same leak that I observe in production, but I would
> like to understand what this one is. 
> This is the server code:
>
> Server.py:
>
> import twisted.protocols.basic
> from twisted.internet.protocol import Factory
> from twisted.internet import reactor
>         
> class HiRate(twisted.protocols.basic.LineOnlyReceiver):
>         MAX_LENGTH = 20000
>         
>         def lineReceived(self, line):
>                 if line == 'get':
>                         out = 'a'*4000+'\r\r' 
>                         self.transport.write(out)
>         
>                 
>                 
> factory = Factory()
> factory.protocol = HiRate
>         
> reactor.listenTCP(8007, factory, backlog=50, interface='10.18.0.2')
> reactor.run()   
> [...]
> To reproduce the memory leak, I either need two machines with fast LAN
> between them (since the client program takes 100% CPU), or possibly one
> machine with a dual core CPU (I have not tried that). It is important that
> client.py is given a separate CPU to run.
> When the length of the response from the server is sufficient,  (out =
> 'a'*4000+'\r\r' ,  4000 is enough in my case), the RSS of the server process
> starts to leak without a bound.
> If you introduce a small delay in the client (#time.sleep(.001)), the leak
> does not occur.
>
>   
Hi Alec,
I wouldn't call that a memory leak. In your example  ( SC/s = (n* CS) / 
s ;  n > 1, SC: bytes sent to the client, CS: bytes sent to the server ) 
there is some CS that satifies the condition SC/s > [available 
bandwidth]. For all CS that exceed that limit, the data will be first 
queued in the Send-Q of the socket and then inside twisted, because the 
Send-Q is full. That's the reason why you see increasing memory usage, 
but it's not a leak, it is a server that can not cope with this kind of 
DOS attack. Twisted can not handle this situation for you because that 
would mean it had to decide which packets, connections, etc. to drop. 
Such decisions have to be made by your application.

regards,
Johann