[Twisted-Python] How to profiling twisted web?

[Reza Lotun]

Hi,

> Currently I'm running twisted web via this command line:
> twistd.bat web -n --path=%path%
> ...
> 1. How can I turn on the --profile option?

You should just be able to do something like:

    twistd.bat web -n --path=%path% --profile=stats_obj
--profiler=cProfile --savestats

Then after a duration of time pounding the server with requests you can stop
it and introspect the stats_obj file:

python
>> import pstats
>> s = pstats.Stats('stats_obj')
>> s.sort_stats('time').print_stats()

This will give you an ordered list of functions where the most time has been
spent. It's also useful to sort by cumulative time. For more information on
how to introspect stats objects see the Python documentation for the pstats
and cProfile module.

> 2. Any way to know how many millisecond elapsed between "request received"
> and render_POST call?

You'll have to measure this yourself I think. A glance at the twisted
codebase informs me:
- In twisted/web/http.py incoming requests are handled by an
HTTPChannel (the protocol from the http server factory)
- Incoming data is processed by the lineReceived method which creates
a Request object
- When all content for a request is received the allContentsReceived
method is invoked, which in turn calls the requestReceived method on
the Request object, which does some internal processing like parsing a
POST-encoded body if there is one. This then calls the process method
on the Request object
- In twisted/web/server.py the Request subclass' process method then
finds the resource being requested and calls the render method

I imagine you could modify the code in the above places to set global
variables by calling time.time(), and dumping them out at some point.

> 3. If there are a bunch of files need to be uploaded (say at 50MB/s speed )
> and save by server, Should I write the file directly in render_POST or using
> deferred call?

Do you mean the server will be *receiving* a file in the HTTP request
body? You can access the request body by looking at request.content
which exposes a File-like API (that is, threat it like you would an
open file in Python).

If you want to *send* a file via Twisted, take a look at:
http://twistedmatrix.com/documents/current/core/howto/producers.html
and
http://twistedmatrix.com/documents/9.0.0/api/twisted.protocols.basic.FileSender.html

Cheers,
Reza

-- 
Reza Lotun
mobile: +44 (0)7521 310 763
email:  rlotun at gmail.com
work:   reza at tweetdeck.com
twitter: @rlotun