[Twisted-Python] LoopingCall at a non-idling reactor

[Jean-Paul Calderone]

On Sun, 19 Jul 2009 17:20:46 +0400 (MSD), Ilya Etingof <ilya at glas.net> wrote:
>
>>> But why main loop does not fire timed events when it has a chance to do
>>> that? Note that my data processor takes 0.2 sec on each run, while my
>>> timed event period is 1 sec.
>>>
>>> In other words, I'd understand this behavior if my data processor
>>> would block main loop for a few periods of timed event, but this
>>> is not the case.
>>
>> Because Twisted receives as many datagrams as possible before going back
>> round the select loop (I think). So, 10 calls to datagramReceived are
>> done (taking 10*0.2 seconds) before twisted gets a chance to schedule
>> any other things.
>
>Perhaps it works that way. And what makes things worse is that if datagram
>input rate is steady and higher than datagram processing time, the
>LoopingCall calls will never be invoked.

It will eventually stop reading datagrams and go do something else.  The
exact way it decides when to stop is completely arbitrary and I don't
think anyone has ever demonstrated that it's clever or appropriate in the
general case (it stops after reading 256k of datagrams).

>
>> As JP has said, you're not allowed to block the main loop like that, but
>> I've seen similar problems with even relatively quick datagramReceived
>> handlers (~0.1msec) under very high load (1000pps) resulting in a form
>> of starvation. This isn't generally a problem with TCP calls due to flow
>> control.
>
>I do not think I'm blocking the main loop so much. I'd say I create a
>condition when data processing time is a little bit longer than the period
>of incoming datagrams arrival.
>
>By way of feedback, it looks to me that if reactor API would give
>user some degree of control on timed versus receiption calls sequencing,
>that would become a more-or-less elegant solution for issues like mine.

One possibility is that we could document the limit I described above and
tell people to adjust it upwards or downwards as they so desire.

>
>> The solution is:
>>
>>  def datagramReceived(...):
>>    reactor.callLater(0, dorealwork, ...)
>>
>>  def dorealwork(self, ...):
>>    ...
>>
>> This will schedule the "real" work as soon as possible in the next
>> reactor loop, but also "fairly" in line with other calls.
>
>Thanks for the hint!
>
>With this callLater approach, timer function works better, though, still
>not absolutely fair:
>
> [snip]
>
>As we can see, timer call period seems to grow over high load.
>

reactor.callLater(0, f) isn't magic.  I probably wouldn't have suggested
it as a solution here.  All it means (more or less) is "do this work after
all current i/o events have been dispatched."  You still might end up with
more than one second of processing bunched up together, and so you'll still
miss a LoopingCall iteration.

Jean-Paul