[Twisted-Python] Reactor callback from the wrong thread

[Lars Ivar Igesund]

Hi!

I have an issue where the reactor calls the callback from a different
thread than the one the reactor is running in.

The usecase is as follows;

We are running tests using PyFit (Fitnesse), and where some of the
tests depends on receiving SNMP traps in the background. So at the
start we start a trap deamon defined as

class TrapDeamon(Thread, netsnmp.Session)

We use the pynetsnmp.twistedsnmp package and has a run() method that
looks like this

def run(self):
    self.awaitTraps(self.hostPort)
    twistedsnmp.updateReactor()
    reactor.run(installSignalHandlers=0)

In the callback from the reactor, we get the pdu of the trap which is
then sent into a trap buffer. On the PyFit side of things, we execute
some commands, and at some points we need a confirmation from a trap
before we can continue. This is done by waiting on a queue
(Queue.Queue). The trap buffer has a register over queues and awaited
trap OID's, and when the correct one is added to the buffer from the
reactor callback, it is put onto the queue, the test rejoice, and
continues.

However, I noticed that fairly often (say every 3rd wait), the wait
timed out even though I had seen the trap on wireshark. So I improved
the logging, and noticed that in the cases where the wait timed out,
the reactor callback for the trap I waited for happened in the main
thread (at least main thread id is reported by python logging), and
thus did not appear until immediately after the wait had failed and
unblocked. Most of the trap callbacks (both before and after these
misses) happens in the correct (reactor) thread though.

What can cause this and how may I debug it? AFAIK, PyFit does not use
twisted or threading at all, and we only have two simple background
threads our selves in addition to the one running the trap deamon.

Best regards,
Lars Ivar Igesund