[Twisted-Python] spawnProcess - reapProcess not retrying on failures

[exarkun at twistedmatrix.com]

On 06:30 am, adi at roiban.ro wrote:
>On 4 September 2014 12:59,  <exarkun at twistedmatrix.com> wrote:
>[snip]
>>What would be really nice is if someone collected *all* of the 
>>complaints
>>about `spawnProcess` into one place and integrated solutions to them 
>>into a
>>design for a replacement. :)
>>
>>Jean-Paul
>
>Since I can not create wiki pages here are my notes:
>
>To do:
>
>* Always return a deferred. #2415
>
>* Provide the option to timeout the execution... and on timeout errback 
>with
>  a dedicated failure #2415
>
>* Add childs to the same process group #2415
>
>* allways errback when failing to spawn (rather than raising OSError) 
>and
>  errback with a different failure when failing to spawn the executable 
>#4184
>
>* on Unix provide a method to install SIGCHLD handler, independent of 
>the
>  general reactor.run() signal handlers. ... maybe related to #5710
>
>* Allow passing Unicode environment (at least on Windows) maybe needed
>for Python3 #5968
>
>* Allow passing Unicode command and arguments on Windows (current not
>supported by pywin32 CreateProcess) #6470... maybe needed for Python3
>#5968
>
>* Improve fork/exec for speed and memory usage #5710 .... maybe by 
>creating
>  a auxiliary process which handles forks and child management.
>
>* Execute as a different account in Windows
>
>* Spawn only after reactor starts ? I am ok with forking before.
>
>* Support client endpoint?
>
>-------
>
>For a new design:
>
>* I like the ProcessProtocol API

I don't. :)  ProcessProtocol means you can't re-use any of your existing 
IProtocol implementations.  Oops.

It seems like a better API would let you say "Here is a protocol, hook 
its output up to fd 0 in the process, hook fd 1 in the process up to its 
input.  Here is another protocol, hook fd 2 in the process up to its 
input" (this would probably be a common configuration - "speak some 
regular protocol over stdin/stdout, have a little special logic 
(probably logging) for stderr).

On the other hand, it's possible to build this on top of ProcessProtocol 
and no one ever has...
>* spawnProcess will always return a deferred... for backward 
>compatibility
>  we need to decide a name for this new method which returns a deferred.
>  Maybe createProcess or connectProcess
>
>* create spawnUnixProcess which has childFDs uid/gid
>
>* create spawnWindowsProcess which has Windows specific options
>
>* createProcess / connectProcess will call spawnUnixProcess or
>spawnWindowsProcess
>  depending on current OS
>
>* ProcessProtocol.processEnded is similar to connectionLost..
>  so maybe it can be rename it and add other methods so that 
>IProcessProtocol
>  can inherit from IProtocol. I guess that this will help with 
>connectors.

Let's not introduce new cases where inheritance is encouraged.
>------
>
>I think that child management problem, does not affect the public API 
>so I am
>not discussing it here.
>
>Timeouts are handled at protocol level and I see that we have 2 timeout 
>types:
>total execution time and  timeout after last child data was received.
>
>-----
>
>Does it make sense to put it on wiki or what should be the Twisted
>Enhancement Proposal process?

I think it's easier to maintain a document on the wiki than in threads 
on the mailing list.
>Can I get 'create page' rights on Trac?

Hm.  I thought everyone had wiki edit rights (except for a small number 
of privileged pages) by default - so I'm not sure what's going on here 
or how to give you permissions.  Perhaps something changed in the trac 
upgrade or as part of the recent ticket permission reconfiguration.

Jean-Paul