[Twisted-Python] spawnProcess help needed

[Jean-Paul Calderone]

On Tue, 9 Dec 2008 15:43:00 -0500, Phil Christensen <phil at bubblehouse.org> wrote:
>Hello All-
>
>I'm running into an issue with a project I'm working on that involves 
>coordinating a large number of external processes to encode audio into 
>different formats, and do other bookkeeping.
>
>I seem to be having a problem when I use spawnProcess to call the  md5sum 
>binary. The processes spawned seem to stick around forever,  until I kill 
>the server. This doesn't appear to happen with any other  binary.
>
>I start with the following abstract class:
>
> [snip MD5Checksum ProcessProtocol]
>
>With the plan of executing it inside an inlineCallbacks-decorated  function, 
>like this:
>
>         proto = yield process.MD5Checksum.run(filename)
>         checksum = proto.get_checksum()
>
>Everything is basically working, except for the fact that the md5sum 
>processes never go away. It does seem that processEnded is being  called, 
>since that's what issues the callback on the deferred returned  by run(), 
>but the process sticks around, as a non-zombie, non-defunct  process that 
>still appears to be using small amounts of CPU time.

processEnded shouldn't be called until the child has exited and been
reaped.  This means that the md5sum process should not still exist in
any form if processEnded is being called.  So either the processes you're
seeing aren't associated with the ProcessProtocol instances you think, or
there's a pretty serious bug in Twisted. :)

A self-contained example of this behavior would be pretty handy.  You should
probably start by making sure processEnded is really being called for the
ProcessProtocol instances associated with the processes which aren't actually
exiting.

>
>Also, once the checksum has been retrieved, killing the server doesn't 
>cause any errors, unlike when a process is legitimately terminated by 
>killing the server, which displays an error (since right now i'm not 
>actually catching ProcessTerminated scenarios):
>
>     2008-12-09 15:17:48-0500 [-] Unhandled error in Deferred:
>     2008-12-09 15:17:48-0500 [-] Unhandled Error
>         Traceback (most recent call last):
>         Failure: twisted.internet.error.ProcessTerminated: A process  has 
>ended with a probable
>     error condition: process ended by signal 2.
>
>Any help in this matter would be appreciated...
>

Jean-Paul