[Twisted-Python] serializing inline callbacks

[Axel Rau]

I have a web application, which runs several subprocesses to build a page.
This runs pretty well with subprocess.check_output(…).
To improve concurrency with other web sessions, I’m converting this to
ireactorprocess.spawnProcess with inline callbacks:
—-
class SubprocessProtocol(ProcessProtocol):
    outBuffer = ""
    errBuffer = ""

    def connectionMade(self):
        self.d = Deferred()

    def outReceived(self, data):
        self.outBuffer += data

    def errReceived(self, data):
        self.errBuffer += data

    def processEnded(self, reason):
        if reason.check(ProcessDone):
            self.d.callback(self.outBuffer)
        else:
            self.d.errback(reason)

def subprocess(args, ireactorprocess=None):
    """
    :type args: list of str
    :type ireactorprocess: :class: twisted.internet.interfaces.IReactorProcess
    :rtype: Deferred
    """
    deblog('Running ' + str(args))
    if ireactorprocess is None:
        from twisted.internet import reactor
        ireactorprocess = reactor

    pprotocol = SubprocessProtocol()
    ireactorprocess.spawnProcess(pprotocol, args[0], args)
    return protocoled
—-
I have chained up the calls to subprocess with the expectation, that they will
be called sequentially one after the previous has been finished:
—-
    def render_GET(self, request):
        ...
        self.A()
        return NOT_DONE_YET
        
    @inlineCallbacks
    def A(self):
        ...
        tmp = yield subprocess(args)
        ...
        self.B()
    
    @inlineCallbacks
    def B(self):
        ...
        tmp = yield subprocess(args)
        ...
        self.C()

    def C(self):
        ...
        x = self.D(...)
        y = self.D(...)
        z = self.D(...)
        ...
        self.E()

    @inlineCallbacks
    def D(self):
        ...
        tmp = yield subprocess(args)
        ...
        returnValue(tmp)

    
    @inlineCallbacks
    def E(self):
        ...
        tmp = yield subprocess(args)
        ...
        request.write(...)
        request.finished
—-
In C, the three calls to self.D run in parallel.
How can I serialize this?


Thanks, Axel
---
PGP-Key:29E99DD6  ☀ +49 151 2300 9283  ☀ computing @ chaos claudius