[Twisted-Python] How to debug an AMP connection?

[Peter Westlake]

Hello,

I'm having trouble with an AMP connection that doesn't fire
the Deferred returned by callRemote. The AMP command copies
files from the client (called a "worker" in the code below)
to a server ("controller"). It sends a chunk of text at a
time - 16K originally, but I tried smaller amounts too.
The server appends the text to a queue to be written;
it shouldn't take long, because it doesn't do the I/O
synchronously, just starts a thread.

What happens is that the client calls callRemote, but the
Deferred that callRemote returns is never fired. I know
the command reaches the server, because the client keeps
track of how many bytes have been sent successfully, and
the file on the server is bigger than that. I know that
the Deferred doesn't fire, because I can examine it using
a manhole and it has called = False.

There are no error messages in the log. That's consistent
with the Deferred not firing, because the error callback
(self.next_or_retry) would log a message if the remote
command failed.

So, what can possibly be happening?
And how can I find out?

All suggestions most gratefully received.

Peter.


Client-side code:

class ConnectionBuffer(object):
    """Class to store AMP commands and retry them after network outages.
    ...
    """

    def send(self, adv_arg, *a, **kw):
        if self.protocol:
            # call_d is here purely to make the Deferred accessible when
            debugging.
            self.call_d  = self.protocol.callRemote(*a,
            **kw).addBoth(self.next_or_retry, adv_arg)

    def connected(self, protocol):
        self.protocol = protocol
        self.send_next()

    def disconnected(self):
        self.protocol = None


Server:

class ControllerProtocol(amp.AMP):
    """Protocol used by the controller to communicate with a worker.
    ...
    """

    def logoutput(self, product, branch, site, job, action, number,
    data, stream):
        """Receive stdout or stderr from an action running on a
        worker."""
        actid = (product, branch, site, job, action)
        ac = self.factory.controller.actions.find(actid)
        ac.instances[number].logfile[stream].write(data)
        return {'status': True}
    commands.LogOutput.responder(logoutput)