[Twisted-Python] thoughts on a reactor feature

[Bruce Mitchener]

Hey all,

When working with Twisted, an important thing is to make sure that the 
reactor is running often enough and that tasks are not starving the 
reactor.  (This is obvious.)

I think it'd be nice if the reactor supported some interface for 
discovering, diagnosing and debugging long running tasks.  This could be 
done in a few different ways.

One would be have an interface to POSIX clocks and track the amount of 
time spent between calls out of reactor.run() and report anything that 
takes over a specified threshold of time to execute.

Another is to implement some sort of stack trace sampling where you 
periodically take a stack trace when a utility program is running and 
you've noted periods of long latency.

I know how to do that sort of thing in C or C++, but not in Python.  Is 
it possible? Efficient enough to actually do?  Would it need some native 
code for high enough resolution clocks?

Much of this would also provide the support needed for something like a 
top-for-reactor as well or to be able to print out all currently queued 
activities.

Information about what each thread in the thread pools are doing would 
be useful as well.  This would match the sort of functionality that DBs 
provide to diagnose their behaviors and what they're currently doing 
with each thread.

For some information on POSIX clocks, see this:

http://www.opengroup.org/onlinepubs/007904975/functions/clock_settime.html

The _POSIX_CPUTIME and _POSIX_THREAD_CPUTIME extensions are particularly 
useful for this sort of thing.  This would (obviously) have to have 
platform dependencies.  The information on the Linux glibc compliance 
with the various POSIX options can be found at:

http://people.redhat.com/drepper/posix-option-groups.html

So, for Linux people, this should all work.  Other platforms would 
either need to have similar support for the specifications, an alternate 
API with an interested implementor, or would need to just not offer the 
additional capability.

Thoughts? Better or simpler ways to do this?

  - Bruce