[Twisted-Python] Question about "starving" the reactor

Luke Marsden luke-lists at hybrid-logic.co.uk
Mon Jun 20 12:23:41 EDT 2011


On Mon, 2011-06-20 at 11:09 -0400, Glyph Lefkowitz wrote:
> On Jun 20, 2011, at 7:39 AM, Phil Mayers wrote:
> > On 20/06/11 11:29, Orestis Markou wrote:
> >> It might be there's no way to do guarantee something like that, so it
> >> might be just that everyone should be careful about this.
> > 
> > I don't think you can stop callers of a function doing silly things with 
> > the return value.
> 
> +1.  The best way to deal with this is to make your APIs nice and
> simple, and their implementation straightforward.  If you try to do
> weird tricks to make callbacks on your Deferreds cooperative, then you
> might break an otherwise reasonable strategy on the part of the client
> code to be well-behaved on their part.
> 
> For the code that is itself trying to be well-behaved, there are
> things like twisted.internet.task.cooperate.

There might not be any way to stop consumers of your Deferreds executing
blocking operations, but when trying to track down which pieces of code
are the culprits, I found exarkun's BigTimesliceTimer very useful:

http://twistedmatrix.com/trac/browser/sandbox/exarkun/btt.py

It uses a reactor-independent timing mechanism (the setitimer sigcall on
Linux, contact me if you want the FreeBSD version) to set an alarm which
the reactor then has to "race" to unset, else the itimer handler gets
pre-emptively executed.  When the alarm runs, it prints a traceback of
the current execution point in the "client" code.  By setting the alarm
frequency sufficiently low and by watching for sufficiently long you can
"sample" the code which is running while the reactor is blocked.  Code
paths which show up frequently are therefore *more likely* to be the
culprits.

One day it would be nice to turn this into some kind of statistical tool
for highlighting which code paths are the "hot-spots" in your code, so
that you can optimise the "blockiest" bits first.  Premature
optimisation, etc.

Hope this helps; it helped me :-)

-- 
Best Regards,
Luke Marsden
CTO, Hybrid Logic Ltd.

Web: http://www.hybrid-cluster.com/
Hybrid Web Cluster - cloud web hosting

Phone: +447791750420




More information about the Twisted-Python mailing list