[Twisted-Python] profiling live twisted application

[Alec Matusis]

> What I did for our web app is add an option to log the time it took to
> handle each page fetch. By looking at the log, you can spot pages that
> are always slow, pages which are slow for certain inputs and pages
> which vary in speed.


Unfortunately we need more detailed results than this. The bottlenecks are in our code, we would like to know which functions take most time, or even which loop within a function. 

The set of client commands to our TCP server is rather large, and the order of these commands is random, so I modeling in dev environment involves potentially wrong pre-assumptions about which commands/functions are most CPU intensive.

It would be nice to at least get the list of functions with total times spend in each function after say 1 min or server run.

> -----Original Message-----
> From: twisted-python-bounces at twistedmatrix.com [mailto:twisted-python-
> bounces at twistedmatrix.com] On Behalf Of Maarten ter Huurne
> Sent: Thursday, November 08, 2007 3:40 PM
> To: Twisted general discussion
> Subject: Re: [Twisted-Python] profiling live twisted application
> 
> On Thursday 08 November 2007, Alec Matusis wrote:
> 
> > What is the best non-intrusive way to profile a live twisted
> application?
> 
> I'm not sure it's possible to do profiling in a non-intrusive way: the
> more detailed measurements you want to take, the more you're going to
> slow down execution. I hear quantum mechanics has similar problems ;)
> 
> > We have TCP servers with epoll reactors.
> >
> > We want to optimize them as our usage grows.
> 
> What I did for our web app is add an option to log the time it took to
> handle each page fetch. By looking at the log, you can spot pages that
> are always slow, pages which are slow for certain inputs and pages
> which vary in speed.
> 
> There are several profilers available for Python, but their overhead is
> probably too much to use in a live environment. Maybe you can duplicate
> the data sets and requests from the live environment in a test
> environment though and profile your server there.
> 
> Here is some example code:
> 
> if _timeRender:
> 	from time import time
> if _profileRender:
> 	from hotshot import Profile
> 
> [...]
> 
> 	if _timeRender:
> 		start = time()
> 	if _profileRender:
> 		profile = Profile('request.prof')
> 		profile.runcall(page.respond, req)
> 		profile.close()
> 	else:
> 		page.respond(req)
> 	if _timeRender:
> 		end = time()
> 		print 'Responding took %1.3f seconds' % (end - start)
> 
> HotShot profile results are binary files, read the Python library
> documentation to learn how you can create human readable output from
> them.
> 
> Bye,
> 		Maarten