[Twisted-Python] Custom Scheduler

Itamar Turner-Trauring itamar at itamarst.org
Mon Apr 8 19:38:47 EDT 2013

On 04/08/2013 05:10 PM, Gerrat Rickert wrote:
> Some background:
> I have implemented a “relative frequency scheduler”.
> Tasks added to it are run with frequencies relative to one another – 
> basically, if tasks 1-3 have relative frequencies of 1,2, and 3 
> respectively, then task 2 is run 1/2 as often as task 1, and task 3 is 
> run 1/3 as often.
> The tasks are all generators, so each iteration of the scheduler 
> yields one item from one of its scheduled generators.
> The scheduler will also run tasks on both a periodic basis (ie.run 
> this task every X other tasks), as well as running tasks on a timed 
> basis (ie. every X seconds).
> I’d like to use twisted in the same program to do some network I/O as 
> well as database interaction.
> I did a small test with adding a task that calls both: 
> reactor.doIteration(0) and reactor.runUntilCurrent() to my scheduler, 
> and that seems to run both the network I/O, and the database interactions.
> I was wondering if there were other issues with this type of setup?
> I can’t find any examples online of using other schedulers with twisted.
By its nature, you cannot use a different schedule with an event loop. 
The event loop must drive things. So you need to use reactor.callLater 
to drive this if you want to work with Twisted in the same process. (Or, 
use http://github.com/itamarst/crochet, soon to be released, but that's 
more of a last resort).

That being said, the interface can be implemented by things other than a 
reactor, e.g. twisted.internet.task.Clock which is mostly used for unit 
tests. So you can have code that works both with and without Twisted if 
you write your code against the callLater API 
(twisted.internet.interfaces.IReactorTime is the interface I think).

> I glanced briefly at t.i.task.Cooperator, but I don’t really need 
> deferreds for any of the CPU stuff, and I’m not really sure how I 
> could replace its scheduler with a non-reactor based one.
Cooperator is basically a class for driving things based off iterators, 
including generators, so it's actually fairly close to what you're doing.

More information about the Twisted-Python mailing list