[Twisted-Python] twisted.spread PB over a ProcessProtocol
Thomas Vander Stichele
thomas at apestaart.org
Sat Aug 15 08:36:13 EDT 2009
for a project I'm working on I wanted to create a helper program which
can spawn an instance of itself as a separate process, slaving itself to
the spawner, and speaking PB to it.
The idea is that the master can spawn processes which will run code that
might segfault or otherwise behave badly (like consume too much memory).
I'm planning to use this as a job server for a django site, where the
spawned jobs then do all kind of media analysis (acoustical
fingerprinting, tag reading, ...)
The jobs themselves however I simply want to write in python, defining
them as methods on a slave-side pb object that the master process can
I prefer to go with spawnProcess and ProcessProtocol because that makes
cleanup easier - no sockets or anything to deal with.
I made a first implementation here:
Basically, I tried to respect Twisted's setup for protocols and
factories and such, providing a listenProcess and connectProcess method
much like the usual ones.
That would mean, I think, that this master/slave abstraction and the
ProcessProtocol could be used by any kind of server/client factory, not
Currently, I have the processprotocol creates process pipes instead of
using stdin/stdout. I'm guessing that would make it
windows-incompatible, correct ? I might have to fall back to using
stdin/stdout, which would make the debugging harder.
a) anyone want to take a look at the code and comment ? Does it make
sense to do it like this ? Any obvious problems ?
b) as the comment at the bottom states, uncommenting that line gives me
a ProtocolError that I cannot figure out. It is repeatable; however,
shuffling code around can make it go away (in fact, I had it in some
other location as well at some point). I'm not sure what would be
triggering that, any ideas ? I am probably doing something dirty in some
part of the code.
c) on hitting ctrl-c, tracebacks are generated, due to this block of
# At least on OS X 10.4, exiting while stdout is non-blocking
# result in data loss. For some reason putting the file
# back into blocking mode seems to resolve this issue.
The exception is EBADF.
That code looks suspicious to me - shouldn't it be at least checking for
exceptions there ? the fd could already be closed IMO.
I love the way you love
but I hate the way
I'm supposed to love you back
URGent, best radio on the net - 24/7 !
More information about the Twisted-Python