[Twisted-Python] How to profiling twisted web?
wanliyou at gmail.com
Mon Dec 28 21:21:39 EST 2009
Thanks for the detailed reply.
twistd.bat web -n --path=%path% --profile=stats_obj --profiler=cProfile
is NOT working for twisted 9.0, --profile and --profiler are both not
Sounds like I start to work until all the data are received, as the content
are multipart/mixed typed pretty large content, I wanna start to find
boundaries and save part of them ASAP. Any clue?
Not only one file, but many files in a multipart/mixed typed body which
totally reach a 50MB/s speed.
I made a tests which doing a multipart/mixed request, containing 50000 * 1KB
small files. Assume socket.sendall(data) is called
render_POST start: 00:03
Yes, it takes 13s to get it done. I can only make the
process_multipart_content spend less time by combining those small files in
several larger files but I still don't know why 3s are spend before
On Tue, Dec 29, 2009 at 2:02 AM, Reza Lotun <rlotun at gmail.com> wrote:
> > Currently I'm running twisted web via this command line:
> > twistd.bat web -n --path=%path%
> > ...
> > 1. How can I turn on the --profile option?
> You should just be able to do something like:
> twistd.bat web -n --path=%path% --profile=stats_obj
> --profiler=cProfile --savestats
> Then after a duration of time pounding the server with requests you can
> it and introspect the stats_obj file:
> >> import pstats
> >> s = pstats.Stats('stats_obj')
> >> s.sort_stats('time').print_stats()
> This will give you an ordered list of functions where the most time has
> spent. It's also useful to sort by cumulative time. For more information on
> how to introspect stats objects see the Python documentation for the pstats
> and cProfile module.
> > 2. Any way to know how many millisecond elapsed between "request
> > and render_POST call?
> You'll have to measure this yourself I think. A glance at the twisted
> codebase informs me:
> - In twisted/web/http.py incoming requests are handled by an
> HTTPChannel (the protocol from the http server factory)
> - Incoming data is processed by the lineReceived method which creates
> a Request object
> - When all content for a request is received the allContentsReceived
> method is invoked, which in turn calls the requestReceived method on
> the Request object, which does some internal processing like parsing a
> POST-encoded body if there is one. This then calls the process method
> on the Request object
> - In twisted/web/server.py the Request subclass' process method then
> finds the resource being requested and calls the render method
> I imagine you could modify the code in the above places to set global
> variables by calling time.time(), and dumping them out at some point.
> > 3. If there are a bunch of files need to be uploaded (say at 50MB/s speed
> > and save by server, Should I write the file directly in render_POST or
> > deferred call?
> Do you mean the server will be *receiving* a file in the HTTP request
> body? You can access the request body by looking at request.content
> which exposes a File-like API (that is, threat it like you would an
> open file in Python).
> If you want to *send* a file via Twisted, take a look at:
> Reza Lotun
> mobile: +44 (0)7521 310 763
> email: rlotun at gmail.com
> work: reza at tweetdeck.com
> twitter: @rlotun
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Twisted-Python