[Twisted-Python] LoopingCall at a non-idling reactor
exarkun at divmod.com
Sun Jul 19 09:58:40 EDT 2009
On Sun, 19 Jul 2009 17:20:46 +0400 (MSD), Ilya Etingof <ilya at glas.net> wrote:
>>> But why main loop does not fire timed events when it has a chance to do
>>> that? Note that my data processor takes 0.2 sec on each run, while my
>>> timed event period is 1 sec.
>>> In other words, I'd understand this behavior if my data processor
>>> would block main loop for a few periods of timed event, but this
>>> is not the case.
>> Because Twisted receives as many datagrams as possible before going back
>> round the select loop (I think). So, 10 calls to datagramReceived are
>> done (taking 10*0.2 seconds) before twisted gets a chance to schedule
>> any other things.
>Perhaps it works that way. And what makes things worse is that if datagram
>input rate is steady and higher than datagram processing time, the
>LoopingCall calls will never be invoked.
It will eventually stop reading datagrams and go do something else. The
exact way it decides when to stop is completely arbitrary and I don't
think anyone has ever demonstrated that it's clever or appropriate in the
general case (it stops after reading 256k of datagrams).
>> As JP has said, you're not allowed to block the main loop like that, but
>> I've seen similar problems with even relatively quick datagramReceived
>> handlers (~0.1msec) under very high load (1000pps) resulting in a form
>> of starvation. This isn't generally a problem with TCP calls due to flow
>I do not think I'm blocking the main loop so much. I'd say I create a
>condition when data processing time is a little bit longer than the period
>of incoming datagrams arrival.
>By way of feedback, it looks to me that if reactor API would give
>user some degree of control on timed versus receiption calls sequencing,
>that would become a more-or-less elegant solution for issues like mine.
One possibility is that we could document the limit I described above and
tell people to adjust it upwards or downwards as they so desire.
>> The solution is:
>> def datagramReceived(...):
>> reactor.callLater(0, dorealwork, ...)
>> def dorealwork(self, ...):
>> This will schedule the "real" work as soon as possible in the next
>> reactor loop, but also "fairly" in line with other calls.
>Thanks for the hint!
>With this callLater approach, timer function works better, though, still
>not absolutely fair:
>As we can see, timer call period seems to grow over high load.
reactor.callLater(0, f) isn't magic. I probably wouldn't have suggested
it as a solution here. All it means (more or less) is "do this work after
all current i/o events have been dispatched." You still might end up with
more than one second of processing bunched up together, and so you'll still
miss a LoopingCall iteration.
More information about the Twisted-Python