[Twisted-web] render_GET and memory consumption

[exarkun at twistedmatrix.com]

On 07:02 pm, psanchez at fosstel.com wrote:
>On 10-12-21 12:47 PM, exarkun at twistedmatrix.com wrote:
>>On 04:43 pm, psanchez at fosstel.com wrote:
>>>
>>>OK, I guess I'm being slow :-( Here's another version, same results.
>>>
>>>import os
>>>from twisted.internet import reactor
>>>from twisted.web.server import Site
>>>from twisted.web.resource import Resource
>>>
>>>CHUNK_SIZE = 32*1024
>>>data = os.urandom(10*1024*1024)
>>>chunks = []
>>>
>>>def make_chunks():
>>>      s = 0
>>>      for chunk in iter(lambda: data[s:s+CHUNK_SIZE], ''):
>>>          chunks.append(chunk)
>>>          s = s + CHUNK_SIZE
>>>
>>>class TestPage(Resource):
>>>       isLeaf = True
>>>
>>>       def render_GET(self, request):
>>>           for chunk in chunks:
>>>               request.write(chunk)
>>>
>>>make_chunks()
>>>root = Resource()
>>>root.putChild('test', TestPage())
>>>reactor.listenTCP(8880, Site(root))
>>>reactor.run()
>>
>>This version has flat memory usage on my system - 33MB all the way
>>through.
>>Jean-Paul
>
>Did you try this running the clients on a different machine than the 
>one
>running the server? I find that if I run both the server and the 
>clients
>(using httperf) on the same machine things behave differently. I can
>only see the memory consumption problem consistently when running the
>clients on a separate machine (which would be my real use case).
>
>BTW, I'm running these test on Ubuntu 10.04, Python 2.6.5, twisted
>10.0.0-2ubuntu2.

Argh.  I just ran it over loopback, so I wasn't accurately testing the 
case you described.  James Knight just reminded me that the transport 
will still find a way to copy data in this case, so you're right that 
this still isn't a working solution.  Instead, you have to go all the 
way to producers/consumers, and only write more data to the transport 
buffer when it has finished dealing with what you previously gave it.

Request implements IConsumer, which means you can pass an IProducer 
provider to its registerProducer method.  When the send buffer gets 
close to empty, the IProducer's resumeProducing method will be called 
and you can write some more bytes to the Request.

There are some more complications and subtleties in the 
IProducer/IConsumer interfaces, which you can read about at 
<http://twistedmatrix.com/documents/current/core/howto/producers.html>.

Basically, you'll end up with something like:

    from twisted.python.log import err
    from twisted.protocols.basic import FileSender
    from twisted.web.server import NOT_DONE_YET

    def render_GET(self, request):
        producer = FileSender()
        d = producer.beginFileTransfer(StringIO(chunk), request)
        def finished(ignored):
            request.finish()
        d.addErrback(err, "Streaming data to client failed")
        d.addCallback(finished)
        return NOT_DONE_YET

Sorry for the earlier mis-information.

Jean-Paul