[Twisted-web] render_GET and memory consumption
exarkun at twistedmatrix.com
exarkun at twistedmatrix.com
Tue Dec 21 14:24:20 EST 2010
On 07:02 pm, psanchez at fosstel.com wrote:
>On 10-12-21 12:47 PM, exarkun at twistedmatrix.com wrote:
>>On 04:43 pm, psanchez at fosstel.com wrote:
>>>
>>>OK, I guess I'm being slow :-( Here's another version, same results.
>>>
>>>import os
>>>from twisted.internet import reactor
>>>from twisted.web.server import Site
>>>from twisted.web.resource import Resource
>>>
>>>CHUNK_SIZE = 32*1024
>>>data = os.urandom(10*1024*1024)
>>>chunks = []
>>>
>>>def make_chunks():
>>> s = 0
>>> for chunk in iter(lambda: data[s:s+CHUNK_SIZE], ''):
>>> chunks.append(chunk)
>>> s = s + CHUNK_SIZE
>>>
>>>class TestPage(Resource):
>>> isLeaf = True
>>>
>>> def render_GET(self, request):
>>> for chunk in chunks:
>>> request.write(chunk)
>>>
>>>make_chunks()
>>>root = Resource()
>>>root.putChild('test', TestPage())
>>>reactor.listenTCP(8880, Site(root))
>>>reactor.run()
>>
>>This version has flat memory usage on my system - 33MB all the way
>>through.
>>Jean-Paul
>
>Did you try this running the clients on a different machine than the
>one
>running the server? I find that if I run both the server and the
>clients
>(using httperf) on the same machine things behave differently. I can
>only see the memory consumption problem consistently when running the
>clients on a separate machine (which would be my real use case).
>
>BTW, I'm running these test on Ubuntu 10.04, Python 2.6.5, twisted
>10.0.0-2ubuntu2.
Argh. I just ran it over loopback, so I wasn't accurately testing the
case you described. James Knight just reminded me that the transport
will still find a way to copy data in this case, so you're right that
this still isn't a working solution. Instead, you have to go all the
way to producers/consumers, and only write more data to the transport
buffer when it has finished dealing with what you previously gave it.
Request implements IConsumer, which means you can pass an IProducer
provider to its registerProducer method. When the send buffer gets
close to empty, the IProducer's resumeProducing method will be called
and you can write some more bytes to the Request.
There are some more complications and subtleties in the
IProducer/IConsumer interfaces, which you can read about at
<http://twistedmatrix.com/documents/current/core/howto/producers.html>.
Basically, you'll end up with something like:
from twisted.python.log import err
from twisted.protocols.basic import FileSender
from twisted.web.server import NOT_DONE_YET
def render_GET(self, request):
producer = FileSender()
d = producer.beginFileTransfer(StringIO(chunk), request)
def finished(ignored):
request.finish()
d.addErrback(err, "Streaming data to client failed")
d.addCallback(finished)
return NOT_DONE_YET
Sorry for the earlier mis-information.
Jean-Paul
More information about the Twisted-web
mailing list