[Twisted-Python] LoopingCall at a non-idling reactor
p.mayers at imperial.ac.uk
Sun Jul 19 07:27:46 EDT 2009
On Sun, Jul 19, 2009 at 09:01:34AM +0100, Ilya Etingof wrote:
>[ skipped ]
>> The above is something which you're more or less not allowed to do in a
>> Twisted application. That simulated data processing is blocking the
>> main event loop. Nothing else happens until it finishes - that includes
>> handling more UDP packets and it includes running timed events like the
>> ones LoopingCall sets up.
>But why main loop does not fire timed events when it has a chance to do
>that? Note that my data processor takes 0.2 sec on each run, while my
>timed event period is 1 sec.
>In other words, I'd understand this behavior if my data processor
>would block main loop for a few periods of timed event, but this
>is not the case.
Because Twisted receives as many datagrams as possible before going back
round the select loop (I think). So, 10 calls to datagramReceived are
done (taking 10*0.2 seconds) before twisted gets a chance to schedule
any other things.
As JP has said, you're not allowed to block the main loop like that, but
I've seen similar problems with even relatively quick datagramReceived
handlers (~0.1msec) under very high load (1000pps) resulting in a form
of starvation. This isn't generally a problem with TCP calls due to flow
The solution is:
reactor.callLater(0, dorealwork, ...)
def dorealwork(self, ...):
This will schedule the "real" work as soon as possible in the next
reactor loop, but also "fairly" in line with other calls.
You still shouldn't block the reactor for a "long" time (the exact value
is dependent on your app)
More information about the Twisted-Python