[Twisted-Python] Limiting a task.Cooperator to N work units / sec

exarkun at twistedmatrix.com exarkun at twistedmatrix.com
Fri Oct 1 08:46:24 EDT 2010


On 09:41 am, p.mayers at imperial.ac.uk wrote:
>Is there an easy way to make a task.Cooperator instance only execute N
>ticks / sec, summed across all iterators it's driving? So if you add 
>two
>iterators, they each run at N/2 per sec, 3 at N/3, etc.
>
>It seems like this ought to do it:

Very close!  It took me a while to notice the mistake.
>N = <rate>
>
>def myScheduler(x):
>   # reschedule N times per second
>   reactor.callLater(1.0/N, x)

The scheduler function must not return None!  This confuses the 
scheduling code in cooperator and causes it to run too many things in 
parallel.  This mistake could be detected easily by cooperator and a 
useful exception raised to point out the problem.  Want to file a 
ticket?

Jean-Paul
>class myTermination:
>   def __call__(self):
>     # stop immediately after one iterator.next()
>     return True
>
>myCoop = task.Cooperator(
>   terminationPredicateFactory=myTermination,
>   scheduler=myScheduler,
>)
>
>...but this doesn't seem to work when I try it:
>
>def mytask(name, limit):
>     for i in range(limit):
>         print name, time.time(), i
>         yield i
>
>def running():
>     finish_a = myCoop.coiterate(mytask('a', 40))
>     def start_b():
>         finish_b = myCoop.coiterate(mytask('b', 10))
>     reactor.callLater(6.5, start_b)
>
>def main():
>     reactor.callWhenRunning(running)
>     reactor.run()
>
>if __name__=='__main__':
>     main()
>
>...shows that, once the "b" iterator is added, each iterator is running
>1/sec, rather than the whole cooperator:
>
>a 1285925864.05 0
>a 1285925865.05 1
>a 1285925866.05 2
>a 1285925867.05 3
>a 1285925868.05 4
>a 1285925869.05 5 << running fine 1/sec up until here
>b 1285925870.05 0
>a 1285925870.55 6 << now running 2/sec
>b 1285925871.05 1
>a 1285925871.55 7
>b 1285925872.05 2
>a 1285925872.55 8
>b 1285925873.05 3
>
>I've had a look at the source code, and it looks like the logic should
>cause what I want to happen, but obviously it's not. Version is Twisted
>8.2.0 on python 2.6 (Fedora 12)
>
>_______________________________________________
>Twisted-Python mailing list
>Twisted-Python at twistedmatrix.com
>http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python



More information about the Twisted-Python mailing list