[Twisted-Python] twisted compatibility with multiprocessing module in fork+execv mode
fgrossi at voismart.it
Wed Sep 30 04:25:36 MDT 2015
I know the multiprocessing module is not properly supported by twisted apps because of the interactions among duplicated file descriptors and signal handling, as discussed other times.
But python 3.4 introduces a new mode to use that module by spawning (i.e. fork() followed by execv()) the new processes instead of simply forking it.
So my question is how supported this is by twisted, and in general how safe it is to use subprocesses created by duplicating the parent immediately followed by the execv of a fresh interpreter.
What i'm thinking is something like this, to asynchronously process requests and delegate the cpu-bound work to some processes:
work = q.get()
context = multiprocessing.get_context('spawn')
q = context.Queue()
p = context.Process(target=worker, args=(q,))
q.put_nowait(work) # when async requests are made
if __name__ == '__main__':
from twisted.internet import reactor
More information about the Twisted-Python