[Twisted-Python] profiling live twisted application

[Maarten ter Huurne]

On Thursday 08 November 2007, Alec Matusis wrote:

> What is the best non-intrusive way to profile a live twisted application?

I'm not sure it's possible to do profiling in a non-intrusive way: the more 
detailed measurements you want to take, the more you're going to slow down 
execution. I hear quantum mechanics has similar problems ;)

> We have TCP servers with epoll reactors.
>
> We want to optimize them as our usage grows.

What I did for our web app is add an option to log the time it took to 
handle each page fetch. By looking at the log, you can spot pages that are 
always slow, pages which are slow for certain inputs and pages which vary 
in speed.

There are several profilers available for Python, but their overhead is 
probably too much to use in a live environment. Maybe you can duplicate the 
data sets and requests from the live environment in a test environment 
though and profile your server there.

Here is some example code:

if _timeRender:
	from time import time
if _profileRender:
	from hotshot import Profile

[...]

	if _timeRender:
		start = time()
	if _profileRender:
		profile = Profile('request.prof')
		profile.runcall(page.respond, req)
		profile.close()
	else:
		page.respond(req)
	if _timeRender:
		end = time()
		print 'Responding took %1.3f seconds' % (end - start)

HotShot profile results are binary files, read the Python library 
documentation to learn how you can create human readable output from them.

Bye,
		Maarten
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
Url : http://twistedmatrix.com/pipermail/twisted-python/attachments/20071109/ead574a0/attachment.pgp