[Twisted-Python] Question about processes in python

[Phil Mayers]

On 04/12/2010 01:09 PM, Gabriel Rossetti wrote:
> Hello everyone,
>
> I'd like to run a python function in a new process, like python 2.6's
> multiprocessing does. I wanted to use ProcessProtocol, but it seems to
> only work for executing executables and interacting with them using
> stdin/stdout/other fds. Is there a way of running a python piece of code
> in a new process?

I frequently use a model like this:


#!/usr/bin/python

import sys
# and loads of twisted imports

# function to run in a child process
def work():
     while True:
         cmd = sys.stdin.readline()
         if not cmd:
             break
         # do something
         sys.stdout.write('result\n')
         sys.stdout.flush()

class MyProcessProtocol(protocol.ProcessProtocol):
     def __init__(self):
         self.buffer = ''
         self.queue = []

     def outReceived(self, data):
         self.buffer += data
         while '\n' in self.buffer:
             reply, self.buffer = self.buffer.split('\n', 1)
             d,c = self.queue.pop(0)
             d.callback(reply)

         if self.queue:
             self.transport.write('%s\n' % (self.queue[0][1],)

     def cmd(self, c):
         d = defer.Deferred()
         if self.queue:
             # just enqueue it
             self.queue.append((d,c))
         else:
             # hang onto the deferred and also send the cmd
             self.queue.append((d,c))
             self.transport.write(c+'\n')
         return d

def parent():
     proto = MyProcessProtocol()
     reactor.spawnProcess(
         proto,
        sys.executable,
        (sys.executable, os.path.abspath(__file__), 'WORKER')
     )

     # write some commands to the child
     d = proto.send_cmd('something')
     d.addCallback(result).addErrback(failed)

def twisted_main():
     reactor.callWhenRunning(parent)
     reactor.run()

def main():
     args = sys.argv[1:]
     if args and args[0]=='WORKER':
         # we're a child process
         work()
     else:
         twisted_main()

if __name__=='__main__':
     main()


It's pretty simple; you need to be sure that the module does all the 
imports it needs in the child process. You can extend the above model to 
something more complex than simple line-based results, using e.g. JSON 
or python pickle, to handle exceptions, and so forth.

IMHO Twisted REALLY REALLY needs something built-in to do this. I'm 
aware of Ampoule; it didn't work for me when I tried it (which was a 
while ago) and having to define AMP commands to use it was a bit of a 
turn off for me.