[Twisted-Python] stopping a twistd daemon: SIGTERM or SIGKILL

[Itamar Turner-Trauring]

On Tue, Jul 24, 2012 at 11:18 PM, Tom Sheffler <tom.sheffler at gmail.com>wrote:

> Hi Twisted Folks,
>
> We use monit to start and stop our twistd daemons.  I have set up monit to
> restart some of the daemons when they grow too large.  I use the following
> command:
>
>     /bin/bash -c 'kill -s SIGTERM `cat /var/run/sensrtunw/sensrtunw.pid`'
>
> Occasionally, this is unsuccessful in killing the twistd process, and I
> need to use a SIGKILL manually as a last resort.
>
> Any ideas why?
>
> What do the logs say? You should get a log message saying something like
"Received SIGTERM", anything in the logs after that which indicates what
code is continuing to run?

Threads might be an issue, but not necessarily as a race condition. Do you
see a message about the reactor shutting down ("Main loop terminated"), but
the process is still running? Python processes won't exit if there are
still threads running. So if you have a thread that's doing a long running
database query, the reactor may have stopped, and the thread pool may have
been told to shutdown, but that thread will still be running waiting for a
reply from the database. In this case, unless the database connection is
wedged, simply waiting long enough will let the process exit.

Another thing which might prevent shutdown is having a Service with a
stopService that returns a Deferred; until that Deferred fires the reactor
won't stop. Again, looking at logs should be helpful, since you won't
actually see the final reactor shutdown message ("Main loop terminated.").

-- 
Itamar Turner-Trauring, Future Foundries LLC
http://futurefoundries.com/ — Twisted consulting, training and support.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20120725/019df498/attachment.html>