[Twisted-Python] twisted.web.server and memory
Lenny G Arbage
alengarbage at yahoo.com
Mon Mar 14 01:38:01 EST 2005
A couple of questions about twisted.web server's
1- Is it normal for the server process to consume
large amounts of memory (and never free it) if large
requests are sent?
2- Assuming the answer to #1 is 'no,' what might I be
doing wrong? I have subclassed Resource and added it
to the Site tree with putChild. My Resource subclass
processes mutlipart/form-data style file uploads. It
looks to me like the Request object that is passed in
contains /all/ the data in the request. So if the
client sends a 100MB file, server consumes 100MB of
memory. What I'm seeing is that if I send small file
uploads, server never consumes more than about 30MB
(7MB resident). If I send a single ~100MB file, usage
mushrooms to over 200MB (~140MB resident), and the
memory is never returned to the system.
3- Assuming the answer to #1 is 'yes,' what is the
correct way to architect a solution. Do I need to
subclass web.resource.Resource or web.site.Request to
do my own management of memory (doing partial reads of
the incoming data)? Or is there some other pattern
that I should be using rather than .web.server?
Note that I've run gc with DEBUG_LEAK trying to spot
any problems, and it doesn't report anything.
If a code snippet that reproduces the above would
help, I'd be happy to produce one.
Thanks -- Lenny G.
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
More information about the Twisted-Python