Opened 8 years ago

Closed 5 years ago

Last modified 5 years ago

#2535 enhancement closed wontfix (wontfix)

Allow Twisted to work without interfering with libraries which use SIGCHLD on posix

Reported by: spiv Owned by:
Priority: low Milestone:
Component: core Keywords:
Cc: Branch:
Author: Launchpad Bug:

Description (last modified by glyph)

Twisted currently needs to install a SIGCHLD handler for reactor.spawnProcess to work reliably. This is problematic when Twisted is used with other libraries that depend on handling SIGCHLD in their own way. Currently, the reactor can be run with the installSignalHandlers=False flag to cooperate with such libraries, but then reactor.spawnProcess isn't reliable.

There are a couple of ways to work around this.

  1. When run with installSignalHandlers=False Twisted could use a different strategy for notifications of subprocess termination. spawnProcess could first fork a process that keeps a single file descriptor back to the parent open, and then forks the child that the caller of spawnProcess requested. Then first subprocess could simply do:
  pid = os.fork()
  if pid:
     returnVal = wait(pid)
     os.write(fd, str(returnVal))
     os.close(fd)
     os._exit()

i.e. wait for the child process to finish, and then let the original process know the return value and exit. The original process would know that the subprocess was done when it saw the fd get closed, and would know what the return value was.

This does come at the unfortunate cost of having an additional process (an additional Python process, even!) per running subprocess, which is not insubstantial. However, it is only in a fairly unusual case that you'd need it.

  1. Another option is to repeatedly poll processes, at a timed interval, to see if they've exited yet. Since most processes end up being either very short or very long running, we could probably come up with an heuristic for calculating the next polling interval that wouldn't be too inefficient.
  1. Apparently some reactors (kqueue? epoll? iocp maybe?) may be able to specifically identify process-exited events independent of signals; this may be preferable to listening to SIGCHLD even if we don't need to cooperate with other software which uses it. (Someone else please fill this in, since I'm not familiar with these APIs.)

Attachments (1)

poll-for-exit.diff (895 bytes) - added by glyph 8 years ago.
an very short hack to poll for process exit

Download all attachments as: .zip

Change History (12)

comment:1 Changed 8 years ago by exarkun

Note that as pointed out on #733, it is not necessarily the case that installing a SIGCHLD handler will interfer with the subprocess module (or more commonly popen()).

However, another signal-free solution is just to poll for exited child processes whenever there might be some. This has the advantage of not leaving an extra Python process in memory for every spawnProcess call (not an inconsiderable cost). It has the disadvantage of polling, of course.

Changed 8 years ago by glyph

an very short hack to poll for process exit

comment:2 Changed 8 years ago by glyph

The local hack that I'm using (because pylucene 1.9 and gcj 4.1 collude to stomp SIGCHLD so badly that signal.getsignal can't even tell that it's been stomped) is in poll-for-exit.diff. It is an implementation of exarkun's suggestion to poll.

comment:3 Changed 7 years ago by exarkun

In addition to #733, see #1997 and #791.

comment:4 Changed 7 years ago by glyph

  • Status changed from new to assigned

comment:5 Changed 7 years ago by glyph

  • Summary changed from Allow Twisted to work without SIGCHLD on posix to Allow Twisted to work without interfering with subprocess, popen, and other users of SIGCHLD on posix

comment:6 Changed 5 years ago by glyph

  • Description modified (diff)
  • Owner changed from glyph to exarkun
  • Priority changed from normal to low
  • Status changed from assigned to new
  • Summary changed from Allow Twisted to work without interfering with subprocess, popen, and other users of SIGCHLD on posix to Allow Twisted to work without interfering with libraries which use SIGCHLD on posix

subprocess and popen aren't actually users of SIGCHLD, and as of #733 I don't think we actually conflict with them any more. Updating the summary and description to reflect the actual problem.

I don't think I'm actually interested in attempting this and I don't think it's too high priority, now that the majority of use-cases have been addressed.

comment:7 Changed 5 years ago by glyph

In case it wasn't clear from my last comment: it may be worth addressing anyway, for other reasons. I don't think I'm personally too familiar with those reasons though.

comment:8 Changed 5 years ago by exarkun

  • Owner changed from exarkun to spiv

comment:9 Changed 5 years ago by spiv

  • Resolution set to wontfix
  • Status changed from new to closed

To be honest, I don't really care about more than subprocess and popen, so now that #733 is fixed I'm happy enough.

I think we should just resolve this as wontfix, so I'm doing that now.

I'm certainly not planning to do any work on this. No good solution springs to mind other than adding an API to ask Twisted to leave SIGCHLD untouched and use polling instead (and we already have installSignalHandlers=False).

If someone has a use case for this specific ticket then they can reopen this ticket. Perhaps a specific use case would help inform the solution to this ticket (e.g. a framework for cooperative signal handling? Probably not...).

comment:10 Changed 5 years ago by spiv

I forgot to add the other justification for "wontfix": it's not just that I don't really care, I doubt anyone else does either.

comment:11 Changed 4 years ago by <automation>

  • Owner spiv deleted
Note: See TracTickets for help on using tickets.