[Twisted-web] render_GET and memory consumption
Pedro I. Sanchez
psanchez at fosstel.com
Tue Dec 21 14:53:54 EST 2010
On 10-12-21 02:24 PM, exarkun at twistedmatrix.com wrote:
> On 07:02 pm, psanchez at fosstel.com wrote:
>> On 10-12-21 12:47 PM, exarkun at twistedmatrix.com wrote:
>>> On 04:43 pm, psanchez at fosstel.com wrote:
>>>>
>>>> OK, I guess I'm being slow :-( Here's another version, same results.
>>>>
>>>> import os
>>> >from twisted.internet import reactor
>>> >from twisted.web.server import Site
>>> >from twisted.web.resource import Resource
>>>>
>>>> CHUNK_SIZE = 32*1024
>>>> data = os.urandom(10*1024*1024)
>>>> chunks = []
>>>>
>>>> def make_chunks():
>>>> s = 0
>>>> for chunk in iter(lambda: data[s:s+CHUNK_SIZE], ''):
>>>> chunks.append(chunk)
>>>> s = s + CHUNK_SIZE
>>>>
>>>> class TestPage(Resource):
>>>> isLeaf = True
>>>>
>>>> def render_GET(self, request):
>>>> for chunk in chunks:
>>>> request.write(chunk)
>>>>
>>>> make_chunks()
>>>> root = Resource()
>>>> root.putChild('test', TestPage())
>>>> reactor.listenTCP(8880, Site(root))
>>>> reactor.run()
>>>
>>> This version has flat memory usage on my system - 33MB all the way
>>> through.
>>> Jean-Paul
>>
>> Did you try this running the clients on a different machine than the
>> one
>> running the server? I find that if I run both the server and the
>> clients
>> (using httperf) on the same machine things behave differently. I can
>> only see the memory consumption problem consistently when running the
>> clients on a separate machine (which would be my real use case).
>>
>> BTW, I'm running these test on Ubuntu 10.04, Python 2.6.5, twisted
>> 10.0.0-2ubuntu2.
>
> Argh. I just ran it over loopback, so I wasn't accurately testing the
> case you described. James Knight just reminded me that the transport
> will still find a way to copy data in this case, so you're right that
> this still isn't a working solution. Instead, you have to go all the
> way to producers/consumers, and only write more data to the transport
> buffer when it has finished dealing with what you previously gave it.
>
> Request implements IConsumer, which means you can pass an IProducer
> provider to its registerProducer method. When the send buffer gets
> close to empty, the IProducer's resumeProducing method will be called
> and you can write some more bytes to the Request.
>
> There are some more complications and subtleties in the
> IProducer/IConsumer interfaces, which you can read about at
> <http://twistedmatrix.com/documents/current/core/howto/producers.html>.
>
> Basically, you'll end up with something like:
>
> from twisted.python.log import err
> from twisted.protocols.basic import FileSender
> from twisted.web.server import NOT_DONE_YET
>
> def render_GET(self, request):
> producer = FileSender()
> d = producer.beginFileTransfer(StringIO(chunk), request)
> def finished(ignored):
> request.finish()
> d.addErrback(err, "Streaming data to client failed")
> d.addCallback(finished)
> return NOT_DONE_YET
>
> Sorry for the earlier mis-information.
>
> Jean-Paul
>
> _______________________________________________
> Twisted-web mailing list
> Twisted-web at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Thanks again Jean-Paul,
This one will take me some time to digest and prototype. I'll give it a
whirl to try to implement a solution and I'll post back my results.
--
Pedro
More information about the Twisted-web
mailing list