[Twisted-web] render_GET and memory consumption

Pedro I. Sanchez psanchez at fosstel.com
Tue Dec 21 14:53:54 EST 2010


On 10-12-21 02:24 PM, exarkun at twistedmatrix.com wrote:
> On 07:02 pm, psanchez at fosstel.com wrote:
>> On 10-12-21 12:47 PM, exarkun at twistedmatrix.com wrote:
>>> On 04:43 pm, psanchez at fosstel.com wrote:
>>>>
>>>> OK, I guess I'm being slow :-( Here's another version, same results.
>>>>
>>>> import os
>>> >from twisted.internet import reactor
>>> >from twisted.web.server import Site
>>> >from twisted.web.resource import Resource
>>>>
>>>> CHUNK_SIZE = 32*1024
>>>> data = os.urandom(10*1024*1024)
>>>> chunks = []
>>>>
>>>> def make_chunks():
>>>>       s = 0
>>>>       for chunk in iter(lambda: data[s:s+CHUNK_SIZE], ''):
>>>>           chunks.append(chunk)
>>>>           s = s + CHUNK_SIZE
>>>>
>>>> class TestPage(Resource):
>>>>        isLeaf = True
>>>>
>>>>        def render_GET(self, request):
>>>>            for chunk in chunks:
>>>>                request.write(chunk)
>>>>
>>>> make_chunks()
>>>> root = Resource()
>>>> root.putChild('test', TestPage())
>>>> reactor.listenTCP(8880, Site(root))
>>>> reactor.run()
>>>
>>> This version has flat memory usage on my system - 33MB all the way
>>> through.
>>> Jean-Paul
>>
>> Did you try this running the clients on a different machine than the
>> one
>> running the server? I find that if I run both the server and the
>> clients
>> (using httperf) on the same machine things behave differently. I can
>> only see the memory consumption problem consistently when running the
>> clients on a separate machine (which would be my real use case).
>>
>> BTW, I'm running these test on Ubuntu 10.04, Python 2.6.5, twisted
>> 10.0.0-2ubuntu2.
>
> Argh.  I just ran it over loopback, so I wasn't accurately testing the
> case you described.  James Knight just reminded me that the transport
> will still find a way to copy data in this case, so you're right that
> this still isn't a working solution.  Instead, you have to go all the
> way to producers/consumers, and only write more data to the transport
> buffer when it has finished dealing with what you previously gave it.
>
> Request implements IConsumer, which means you can pass an IProducer
> provider to its registerProducer method.  When the send buffer gets
> close to empty, the IProducer's resumeProducing method will be called
> and you can write some more bytes to the Request.
>
> There are some more complications and subtleties in the
> IProducer/IConsumer interfaces, which you can read about at
> <http://twistedmatrix.com/documents/current/core/howto/producers.html>.
>
> Basically, you'll end up with something like:
>
>      from twisted.python.log import err
>      from twisted.protocols.basic import FileSender
>      from twisted.web.server import NOT_DONE_YET
>
>      def render_GET(self, request):
>          producer = FileSender()
>          d = producer.beginFileTransfer(StringIO(chunk), request)
>          def finished(ignored):
>              request.finish()
>          d.addErrback(err, "Streaming data to client failed")
>          d.addCallback(finished)
>          return NOT_DONE_YET
>
> Sorry for the earlier mis-information.
>
> Jean-Paul
>
> _______________________________________________
> Twisted-web mailing list
> Twisted-web at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web

Thanks again Jean-Paul,

This one will take me some time to digest and prototype. I'll give it a 
whirl to try to implement a solution and I'll post back my results.

-- 
Pedro



More information about the Twisted-web mailing list