[Twisted-Python] Question regarding the working of twisted python with celery

[Andrew Bennetts]

Dinesh Kapoor wrote:
[...]
>    4. When celery is run with concurrency of 2, then I tried printing out the
>    process ids of calling process in the reactor code, and I am getting 2
>    different pids, so I am assuming there are two seperate copies of reactor
>    in those separate tasks.

This sounds like the problem: you have instantiated one reactor, and
Celery (probably via the multiprocessing module) has used fork(), which
shares the reactor's internal state with the forked copy, which is not
safe.  In particular I suspect the 'waker' pipe that the reactor uses to
be notified is being shared between both processes, leading to
notifications of callFromThread etc to be noticed by the wrong copy.

The solution is to avoid starting the reactor until after the separate
processes have started.  I don't know how easy that is to arrange with
Celery.

-Andrew.