[Twisted-Python] process timing / idle handler

[Jp Calderone]

On Wed, 14 Sep 2005 21:31:37 +0200, henning.ramm at mediapro-gmbh.de wrote:
>Ahoi!
>
>I often use reactor.callLater(0, something) to let twisted handle and intercept my call queue.
>Is this really preferable to calling 'something' directly? Or do I just fool myself?
>
>Is there something like reactor.OnIdle()?
>Or, more generally, how could I delay some actions (intermediate cleanup etc.) until the computer has free resources?
>(Some of my processes run on a very high loaded server.)
>

This does what you expect, but what you expect may not be the right thing.  What happens if there are 50 on-idle tasks?  That won't be nearly as nice as if there's only 1.  If you want to schedule CPU-intensive tasks, they need to cooperate with each other.

Take a look at <http://divmod.org/svn/Divmod/trunk/Epsilon/epsilon/cooperator.py>.  It's intended to be used with time-consumed tasks implemented as iterators.  By using a single Cooperator, the amount of work you arrange to have performed per unit of network activity is limited, regardless of the number of different tasks outstanding.

This also lets you tune the rate at which time-consumed tasks are processed using a single knob.  This means that if you notice you can handle greater latency, there is a single thing to change (possibly even at runtime) to let you do this (the benefit being that doing more work at once is more efficient).

You can also nest Cooperators and assign them non-time-based schedulers (such as a system load-based scheduler).  This lets you give a task or group of tasks a priority relative to other things you have scheduled.

However, if you have expensive computations that do not interact with the rest of your application except by receiving an immutable or non-shared chunk of input and producing an immutable or non-shared chunk of output, a threaded solution may well be appropriate.  Threadless solutions are superior because they let the code in question interact with the rest of your application in an unsurprising way.  If there is no interaction, the benefits of avoiding threads are less pronounced.

Jp