[Twisted-Python] LoopingCall at a non-idling reactor
p.mayers at imperial.ac.uk
Sun Jul 19 10:43:58 EDT 2009
On Sun, Jul 19, 2009 at 02:58:40PM +0100, Jean-Paul Calderone wrote:
>It will eventually stop reading datagrams and go do something else. The
>exact way it decides when to stop is completely arbitrary and I don't
>think anyone has ever demonstrated that it's clever or appropriate in the
>general case (it stops after reading 256k of datagrams).
That's interesting. I didn't know that. Has it always done this, or is
that a recent thing?
As I say, I used to experience starvation of the reactor doing SNMP
walks of hundreds of devices (in addition to UDP socket buffer
overflows) which is why I adopted my solution (see below)
>reactor.callLater(0, f) isn't magic. I probably wouldn't have suggested
>it as a solution here. All it means (more or less) is "do this work after
>all current i/o events have been dispatched." You still might end up with
>more than one second of processing bunched up together, and so you'll still
>miss a LoopingCall iteration.
For what it's worth, I actually do something a lot more sophisticated.
My UDP IO, the protocol parsing, re-transmission logic and so forth
actually happen in a "farm" of N sub-processes (which lets me make use
of the multi-core CPU) and I talk to the parent process over TCP.
Viewing the reactor as a device that emits a stream of events (timer
ticks, socket reads, write requests etc.) the problem can be considered
a QoS issue. It's hard to see how the reactor could "know" how to do
the right thing in all cases, absent a hinting mechanism from the
queue = reactor.getDefaultQueue()
subq = queue.newChild()
subq.listenUDP(CONTROL, importantProto(), weight=10)
subq.listenUDP(DATA, lessimportantProto(), weight=1)
...or something ;o) This isn't a serious suggestion obviously!
More information about the Twisted-Python