Allow Twisted to work without interfering with libraries which use SIGCHLD on posix

[spiv]

Twisted currently needs to install a SIGCHLD handler for reactor.spawnProcess to work reliably. This is problematic when Twisted is used with other libraries that depend on handling SIGCHLD in their own way. Currently, the reactor can be run with the installSignalHandlers=False flag to cooperate with such libraries, but then reactor.spawnProcess isn't reliable.

There are a couple of ways to work around this.

1. When run with installSignalHandlers=False Twisted could use a different strategy for notifications of subprocess termination. spawnProcess could first fork a process that keeps a single file descriptor back to the parent open, and then forks the child that the caller of spawnProcess requested. Then first subprocess could simply do:

    pid = os.fork()
    if pid:
       returnVal = wait(pid)
       os.write(fd, str(returnVal))
       os.close(fd)
       os._exit()

i.e. wait for the child process to finish, and then let the original process know the return value and exit. The original process would know that the subprocess was done when it saw the fd get closed, and would know what the return value was.

This does come at the unfortunate cost of having an additional process (an additional Python process, even!) per running subprocess, which is not insubstantial. However, it is only in a fairly unusual case that you'd need it.

2. Another option is to repeatedly poll processes, at a timed interval, to see if they've exited yet. Since most processes end up being either very short or very long running, we could probably come up with an heuristic for calculating the next polling interval that wouldn't be too inefficient.

3. Apparently some reactors (kqueue? epoll? iocp maybe?) may be able to specifically identify process-exited events independent of signals; this may be preferable to listening to SIGCHLD even if we don't need to cooperate with other software which uses it. (Someone else please fill this in, since I'm not familiar with these APIs.)

[exarkun]

Note that as pointed out on #733, it is not necessarily the case that installing a SIGCHLD handler will interfer with the subprocess module (or more commonly popen()).

However, another signal-free solution is just to poll for exited child processes whenever there might be some. This has the advantage of not leaving an extra Python process in memory for every spawnProcess call (not an inconsiderable cost). It has the disadvantage of polling, of course.

[glyph]

an very short hack to poll for process exit

[glyph]

The local hack that I'm using (because pylucene 1.9 and gcj 4.1 collude to stomp SIGCHLD so badly that signal.getsignal can't even tell that it's been stomped) is in poll-for-exit.diff. It is an implementation of exarkun's suggestion to poll.

[exarkun]

In addition to #733, see #1997 and #791.

[glyph]

subprocess and popen aren't actually users of SIGCHLD, and as of #733 I don't think we actually conflict with them any more. Updating the summary and description to reflect the actual problem.

I don't think I'm actually interested in attempting this and I don't think it's too high priority, now that the majority of use-cases have been addressed.

[glyph]

In case it wasn't clear from my last comment: it may be worth addressing anyway, for other reasons. I don't think I'm personally too familiar with those reasons though.

[spiv]

To be honest, I don't really care about more than subprocess and popen, so now that #733 is fixed I'm happy enough.

I think we should just resolve this as wontfix, so I'm doing that now.

I'm certainly not planning to do any work on this. No good solution springs to mind other than adding an API to ask Twisted to leave SIGCHLD untouched and use polling instead (and we already have installSignalHandlers=False).

If someone has a use case for this specific ticket then they can reopen this ticket. Perhaps a specific use case would help inform the solution to this ticket (e.g. a framework for cooperative signal handling? Probably not...).

[spiv]

I forgot to add the other justification for "wontfix": it's not just that I don't really care, I doubt anyone else does either.