[Twisted-Python] twisted.web.server and memory

[Lenny G Arbage]

A couple of questions about twisted.web server's
memory management:

1- Is it normal for the server process to consume
large amounts of memory (and never free it) if large
requests are sent?

2- Assuming the answer to #1 is 'no,' what might I be
doing wrong?  I have subclassed Resource and added it
to the Site tree with putChild.  My Resource subclass
processes mutlipart/form-data style file uploads.  It
looks to me like the Request object that is passed in
contains /all/ the data in the request.  So if the
client sends a 100MB file, server consumes 100MB of
memory.  What I'm seeing is that if I send small file
uploads, server never consumes more than about 30MB
(7MB resident).  If I send a single ~100MB file, usage
mushrooms to over 200MB (~140MB resident), and the
memory is never returned to the system.

3- Assuming the answer to #1 is 'yes,' what is the
correct way to architect a solution.  Do I need to
subclass web.resource.Resource or web.site.Request to
do my own management of memory (doing partial reads of
the incoming data)?  Or is there some other pattern
that I should be using rather than .web.server?


Note that I've run gc with DEBUG_LEAK trying to spot
any problems, and it doesn't report anything.

If a code snippet that reproduces the above would
help, I'd be happy to produce one.

  Thanks -- Lenny G.

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com