[Twisted-Python] How to profiling twisted web?

Wan Li wanliyou at gmail.com
Mon Dec 28 21:21:39 EST 2009

Thanks for the detailed reply.

twistd.bat web -n --path=%path% --profile=stats_obj --profiler=cProfile

is NOT working for twisted 9.0, --profile and --profiler are both not

Sounds like I start to work until all the data are received, as the content
are multipart/mixed typed pretty large content, I wanna start to find
boundaries and save part of them ASAP. Any clue?

Not only one file, but many files in a multipart/mixed typed body which
totally reach a 50MB/s speed.
I made a tests which doing a multipart/mixed request, containing 50000 * 1KB
small files. Assume socket.sendall(data) is called
at 00:00
render_POST start: 00:03
process_multipart_content_start: 00:03
write_all_small_files_in_one_file_start :00:11

Yes, it takes 13s to get it done. I can only make the
process_multipart_content spend less time by combining those small files in
several larger files but I still don't know why 3s are spend before

On Tue, Dec 29, 2009 at 2:02 AM, Reza Lotun <rlotun at gmail.com> wrote:

> Hi,
> > Currently I'm running twisted web via this command line:
> > twistd.bat web -n --path=%path%
> > ...
> > 1. How can I turn on the --profile option?
> You should just be able to do something like:
>    twistd.bat web -n --path=%path% --profile=stats_obj
> --profiler=cProfile --savestats
> Then after a duration of time pounding the server with requests you can
> stop
> it and introspect the stats_obj file:
> python
> >> import pstats
> >> s = pstats.Stats('stats_obj')
> >> s.sort_stats('time').print_stats()
> This will give you an ordered list of functions where the most time has
> been
> spent. It's also useful to sort by cumulative time. For more information on
> how to introspect stats objects see the Python documentation for the pstats
> and cProfile module.
> > 2. Any way to know how many millisecond elapsed between "request
> received"
> > and render_POST call?
> You'll have to measure this yourself I think. A glance at the twisted
> codebase informs me:
> - In twisted/web/http.py incoming requests are handled by an
> HTTPChannel (the protocol from the http server factory)
> - Incoming data is processed by the lineReceived method which creates
> a Request object
> - When all content for a request is received the allContentsReceived
> method is invoked, which in turn calls the requestReceived method on
> the Request object, which does some internal processing like parsing a
> POST-encoded body if there is one. This then calls the process method
> on the Request object
> - In twisted/web/server.py the Request subclass' process method then
> finds the resource being requested and calls the render method
> I imagine you could modify the code in the above places to set global
> variables by calling time.time(), and dumping them out at some point.
> > 3. If there are a bunch of files need to be uploaded (say at 50MB/s speed
> )
> > and save by server, Should I write the file directly in render_POST or
> using
> > deferred call?
> Do you mean the server will be *receiving* a file in the HTTP request
> body? You can access the request body by looking at request.content
> which exposes a File-like API (that is, threat it like you would an
> open file in Python).
> If you want to *send* a file via Twisted, take a look at:
> http://twistedmatrix.com/documents/current/core/howto/producers.html
> and
> http://twistedmatrix.com/documents/9.0.0/api/twisted.protocols.basic.FileSender.html
> Cheers,
> Reza
> --
> Reza Lotun
> mobile: +44 (0)7521 310 763
> email:  rlotun at gmail.com
> work:   reza at tweetdeck.com
> twitter: @rlotun
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

>: ~
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20091229/8b0b27ee/attachment.htm 

More information about the Twisted-Python mailing list