[Twisted-Python] Reactor callback from the wrong thread

[Lars Ivar Igesund]

Hi, thanks for the reply.

On Wed, May 27, 2009 at 5:43 PM, Jean-Paul Calderone <exarkun at divmod.com> wrote:
> On Wed, 27 May 2009 15:08:49 +0200, Lars Ivar Igesund <larsivi at gmail.com> wrote:
>>Hi!
>>
>>I have an issue where the reactor calls the callback from a different
>>thread than the one the reactor is running in.
>
> Generally speaking, the only callbacks the reactor invokes are protocol
> methods (like dataReceived and connectionLost) and timed events (things
> you pass to reactor.callLater).  It always calls these in the thread it
> is running in.  What callbacks are you seeing be invoked in the "wrong"
> thread?

Sorry for being a bit unclear; these callbacks originates from the
SNMP trap listen agent which at the bottom is Net-SNMP wrapped in
Python and a protocol for Twisted (incorporated in pynetsnmp, package
python-pynetsnmp in Ubuntu/Debian).

>
>> [snip]
>>
>>What can cause this and how may I debug it? AFAIK, PyFit does not use
>>twisted or threading at all, and we only have two simple background
>>threads our selves in addition to the one running the trap deamon.
>
> The most likely explanation is that your code (perhaps by way of PyFit,
> I'm not sure -- I've never used PyFit) is calling a Twisted API from a
> thread other than the reactor thread.  So, examine all the places you
> call Twisted APIs (including APIs in twistedsnmp which may call Twisted
> APIs) and make sure they're only run in the reactor thread.

Googling further, it appears that NetSNMP is not considered thread
safe, and this is probably the reason for my troubles (there are
normal netsnmp calls going the other way from the apparently hijacked
thread).

PyFit did not have any twisted dependencies, so I don't consider it an
option at this point.

I have also tested the alternative Twisted SNMP package (based on
PySNMP), but unfortunately both of those appear to mature enough for
our use.

At this point I'm probably left with two options; either make all of
the SNMP communication go via pynetsnmp (will cause at least some
rewriting), or move the trap deamon part into its own application that
communicates with the test framework via a sockets. The latter would
guarantee no snmp thread confusion, but may be a bit overkill.

If anyone has a good advice, then I'd appreciate it.

Thanks!