[Twisted-Python] twisted.spread PB over a ProcessProtocol

[Thomas Vander Stichele]

Hi everyone,

for a project I'm working on I wanted to create a helper program which
can spawn an instance of itself as a separate process, slaving itself to
the spawner, and speaking PB to it.

The idea is that the master can spawn processes which will run code that
might segfault or otherwise behave badly (like consume too much memory).

I'm planning to use this as a job server for a django site, where the
spawned jobs then do all kind of media analysis (acoustical
fingerprinting, tag reading, ...)

The jobs themselves however I simply want to write in python, defining
them as methods on a slave-side pb object that the master process can
call on.

I prefer to go with spawnProcess and ProcessProtocol because that makes
cleanup easier - no sockets or anything to deal with.


I made a first implementation here:

https://thomas.apestaart.org/thomas/trac/browser/tests/twisted/daemon/daemon.py

Basically, I tried to respect Twisted's setup for protocols and
factories and such, providing a listenProcess and connectProcess method
much like the usual ones.

That would mean, I think, that this master/slave abstraction and the
ProcessProtocol could be used by any kind of server/client factory, not
just PB.
Currently, I have the processprotocol creates process pipes instead of
using stdin/stdout.  I'm guessing that would make it
windows-incompatible, correct ? I might have to fall back to using
stdin/stdout, which would make the debugging harder.

Questions:

a) anyone want to take a look at the code and comment ? Does it make
sense to do it like this ? Any obvious problems ?

b) as the comment at the bottom states, uncommenting that line gives me
a ProtocolError that I cannot figure out.  It is repeatable; however,
shuffling code around can make it go away (in fact, I had it in some
other location as well at some point).  I'm not sure what would be
triggering that, any ideas ? I am probably doing something dirty in some
part of the code.

c) on hitting ctrl-c, tracebacks are generated, due to this block of
code:
        # At least on OS X 10.4, exiting while stdout is non-blocking
can
        # result in data loss.  For some reason putting the file
descriptor
        # back into blocking mode seems to resolve this issue.
        fdesc.setBlocking(self.fd)

The exception is EBADF.

That code looks suspicious to me - shouldn't it be at least checking for
exceptions there ? the fd could already be closed IMO.

Comments welcome!

Thomas



-- 
I love the way you love
but I hate the way
I'm supposed to love you back
--
URGent, best radio on the net - 24/7 !
http://urgent.fm/