update/replace spawnProcess and ProcessProtocol

[bigdog]

spawnProcess and Process Protocol has provided many years of service, but it is time to look at twisted mechanism to launch and manage child process execution.

Potential Approach 1) gather requirements (purpose of this ticket) 2) define new API that supports requirements 3) deprecate old spawnProcess and proxcess API

---

1) add the ability to spawn a process on the behalf a specific user.

sample use case: A twisted process is acting as a grid compute node. A user schedules processes to execute on multiple machines. The process mus run as the user requesting the task, not the owner of the twisted grid node process.

Often programs are licensed to specific users

2) ability to add kill timeout to new process .

sample use case: A twisted process is used to validate a product. This validation requires to twisted to spawn a process to run the product being verified. Sometimes the child process "gets lost", and does not terminate. A kill timeout would :

• spawnProcess, register watchdog kill task.
• if the watchdog kill fires before the child process exits, the child process is killed
• the watchdog kill process is disabled when the child process complets

[exarkun]

The first point sounds like adding support for something like uid/gid on posix for win32. This is a good idea. I have no idea how to do it, of course. :P Whatever results, the API should be compatible between posix and windows. That is, uid/gid or username/password should be indirected one layer beyond where it is currently.

The second point I'm not so clear on. It sounds like this is something which should be implemented at the application level and doesn't require any more reactor support than spawnProcess and callLater. Perhaps one thing it does suggest, though, is that it should be easier to use an IProtocol with spawnProcess (which would mean TimeoutMixin could be used with spawnProcess, which would make this case much simpler).

[exarkun]

The current process support is missing these basic things:

• Consistency across platforms
  • Environment behavior differs
  • Different parameters are supported
    • usePTY, uid, gid, childFDs
• An interface defining the protocol which may be used
  • Subclassing ProcessProtocol is basically required
  • The low-level implementation helpers, primarily PTYProcess and Process, call different methods on the protocol
    • Process calls childDataReceived
    • PTYProcess calls outReceived
• Separation between the process exited event and each of the child FD closed events: these are currently all delivered simultaneously once they have all occurred, via processEnded (except closed child FDs may also be delivered sooner)
• Calling reactor.spawnProcess before reactor.run results in an ugly warning: this isn't really an interface problem, just an implementation shortcoming which should be remedied, regardless of the other concerns presented here (but if a new interface is introduced, its implementation should avoid this problem from the start, and if the old interface is re-implemented in terms of the new one, this problem will disappear for it as well)
• PTY support should almost certainly be implemented in terms of childFDs or an equivalent feature, rather than as a flag which completely changes everything.

[bwh]

I would like also like to have some consistency between the interface to a local process and a remote process accessed by Conch (or other mechanism).

I started work on a package that supports spawning of processes in "accounts". I was able to make processes run via SSH appear much like local process, but it required some hacks and I found there was no definition of the process interface. There is IProcessTransport, but local processes don't actually implement that! I implemented most of IProcessTransport and communication with ProcessProtocol but was then interrupted by other work before I could complete the supporting work on accounts and authentication. I should be able to provide the code I have if anyone's interested.

[glyph]

Replying to bwh:

I would like also like to have some consistency between the interface to a local process and a remote process accessed by Conch (or other mechanism). I started work on a package that supports spawning of processes in "accounts". I was able to make processes run via SSH appear much like local process, but it required some hacks and I found there was no definition of the process interface. There is IProcessTransport, but local processes don't actually implement that! I implemented most of IProcessTransport and communication with ProcessProtocol but was then interrupted by other work before I could complete the supporting work on accounts and authentication. I should be able to provide the code I have if anyone's interested.

I am generally sympathetic to this goal, but I think it (and bigdog's initial "watchdog kill" requirement) needs to be kept out of scope for this ticket. One of the major problems with the current interface is that application utility functionality (like processEnded's coalescing of end-of-file and process-terminated events, and IProtocol emulation such as dataReceived, and write) is haphazardly mixed together with core child-process-communication abstractions (like signalProcess, childDataReceived, and writeToChild). The worst consequence of this duplication combined with poor specification is that the cross-platform implementation of some of these features is totally haphazard and broken, only usable from the utility level. Another example, which I'm not sure if there's a ticket for already, is that IProcessTransport is not actually implemented by the process transport on win32 (it's missing writeToChild, among other things).

It's important that we pay close attentinon to separating the reactor-level core functionality here, and the application-level utility interfaces which connect things to processes, or abstract over the distinction between local and remote process spawning.

[bigdog]

Glyph,

I have no problem with placing the responsibility for the watchdog kill on the application. Core twisted provides the tools to easily implement it.

[bigdog]

I have a use case where the spawned process creates a tree of processes.

When I call proc.signalProcess() only the top process is killed. The children of the process happily consume compute resources until manually harvested (or they die).

It would be nice if spawnProcess would have the option to create a process group, and place the spawned process in the process group.

I am not sure about the api for this.

spawnProcess (..... createNewProcGroup=True,..) ??

proc.signalProcessGroup() would kill all processes in the process group. (not sure about api names)

[jknight]

Replying to bigdog:

I have a use case where the spawned process creates a tree of processes. When I call proc.signalProcess() only the top process is killed. The children of the process happily consume compute resources until manually harvested (or they die). It would be nice if spawnProcess would have the option to create a process group, and place the spawned process in the process group.

It would be nice but it's a giant pain in the ass. The only way I know of requires running a separate monitoring process to keep the process-group around. Otherwise, there's no way to reliably send a kill signal after the top process has died, since the process group might've finished, and another with the same pgid started.

[bigdog]

It would be nice to have a seperate callback in the process protocol that would fire when the process was killed via signalProcess().

[bigdog]

Replying to jknight:

Replying to bigdog:

I have a use case where the spawned process creates a tree of processes. When I call proc.signalProcess() only the top process is killed. The children of the process happily consume compute resources until manually harvested (or they die). It would be nice if spawnProcess would have the option to create a process group, and place the spawned process in the process group.

It would be nice but it's a giant pain in the ass. The only way I know of requires running a separate monitoring process to keep the process-group around. Otherwise, there's no way to reliably send a kill signal after the top process has died, since the process group might've finished, and another with the same pgid started.

I don't know about UNIX, but window has the concept of a Job, It would be straightforward to implment this on windows. Thanks for the tip on UNIX, I will research this further.

[bigdog]

Here are a couple of discussions from the twisted mailing list

 http://twistedmatrix.com/pipermail/twisted-python/2005-May/010572.html

 http://twistedmatrix.com/pipermail/twisted-python/2006-July/013693.html

[exarkun]

Supporting process groups/windows jobs would definitely be nice. I don't really know much about windows jobs specifically though. Are the two similar enough to support through the same API?

[teratorn]

Phil Mayers reported on the mailing list about a problem with console windows being created inadvertently on Windows.  http://twistedmatrix.com/pipermail/twisted-python/2007-February/014731.html

I think we need to address these types of issues somehow. There are so many options that you have for spawning processes on Windows that I dread exposing them all through some generic API. Of course we wouldn't do that, but then it leaves anyone who needs those options on their own.

How about if we had one portable API and separate APIs for platform-specific features? E.g. spawnWindowsProcess() which would accept all the arguments that CreateProcess can take.

Are there any crazy *nix process APIs that anyone cares about?

[glyph]

Apropos of #3146, a new process-spawning API should really require the reactor to be running before it actually tries to start the process. This would eliminate the obvious race where the process might end before the reactor starts up to catch the signal.

[glyph]

Replying to teratorn:

How about if we had one portable API and separate APIs for platform-specific features? E.g. spawnWindowsProcess() which would accept all the arguments that CreateProcess can take.

I don't think this would be a good thing to expose in the reactor, since we have certain requirements of, e.g. CreateProcess in our own process-spawning stuff, so we can't allow arbitrary options to be passed.

Some platform-specific hints would be good, in an "extra options" parameter or something; that would allow code to be portable, but suggest certain (explicitly supported by twisted) things on particular platforms.

If you really, really want to call CreateProcess and you need all of its arcane nuances - well, just call CreateProcess. Nobody's stopping you. (In fact, unlike on UNIX you don't even need to deal with an interfering signal handler.)