[Twisted-Python] How to debug an AMP connection?

[Jean-Paul Calderone]

On Fri, 24 Jul 2009 16:12:28 +0100, Peter Westlake <peter.westlake at pobox.com> wrote:
>
>
>On Fri, 24 Jul 2009 10:44 -0400, "Drew Smathers"
><drew.smathers at gmail.com> wrote:
>> On Fri, Jul 24, 2009 at 9:35 AM, Peter Westlake<peter.westlake at pobox.com>
>> wrote:
>> > Hello,
>> >
>> > I'm having trouble with an AMP connection that doesn't fire
>> > the Deferred returned by callRemote. The AMP command copies
>> > files from the client (called a "worker" in the code below)
>> > to a server ("controller"). It sends a chunk of text at a
>> > time - 16K originally, but I tried smaller amounts too.
>> > The server appends the text to a queue to be written;
>> > it shouldn't take long, because it doesn't do the I/O
>> > synchronously, just starts a thread.
>> >
>>
>> "justs starts a thread"? Where are you starting a thread in the code
>> example you've posted?
>
>> Please post a complete example; I don't see methods on client code
>> below such as send_next() or next_or_retry().  It's hard to help
>> without a complete example.
>
>Sorry - I was trying to avoid clogging everyone's inboxes with too
>much code. So far I haven't been able to reproduce the bug in a
>simple way. In the meantime, here's the code that starts the thread.
>It's one of the very first bits of code I wrote with either Twisted
>or Python, so it may do things in quite the wrong way. It serialises
>writes to a file by storing requests in a queue, and it gets called
>by the line
>
>       ac.instances[number].logfile[stream].write(data)
>
>in the command handler.
>
>
>from twisted.internet import defer, threads
>import os, errno
>
>class FileMan(object):
>    def __init__(self, filename):
>        self.filename = filename
>        self.queue = []
>        self.started = False
>
>    def check_file(self):
>        self.started = os.path.exists(self.filename)
>        return self.started
>
>    def makedir(self):
>        dirname = os.path.dirname(self.filename)
>        if not os.path.isdir(dirname):
>            os.makedirs(dirname)
>
>    def add(self, op, *a, **kw):
>        def getnext(result):
>            del(self.queue[0])
>            if len(self.queue) > 0:
>                self.queue[0].callback('unused')
>            return result
>
>        d = defer.Deferred()
>        d.addCallback(op, *a, **kw)
>        d.addBoth(getnext)
>        self.queue.append(d)
>        if len(self.queue) == 1:
>            d.callback('unused')
>        return d
>
>    def write(self, data, mode='a'):
>        def do_write(result, filename, data):
>            def write_in_thread(filename, data):
>                f = open(filename, mode)
>                try:
>                    f.write(data)
>                finally:
>                    f.close()
>                self.started = True
>            d = threads.deferToThread(write_in_thread, filename, data)
>            return d
>        return self.add(do_write, self.filename, data)
>
>
>>  In general, though, if you're trying to
>> debug Deferreds you might try setting debugging with
>> defer.setDebugging(1).
>
>What kind of output does that produce? I did try it,
>but didn't see anything out of the ordinary.

It makes Deferreds keep track of the call stack when they are created and
when they are invoked.  This can be helpful tracking down Deferred failures
that don't otherwise identify themselves very well.  If you're not seeing
any failures being logged, then this probably won't help much, since there
won't be anything to increase the verbosity of. :)

Another thing to check out would be a network capture, to see if the server
is actually sending back a response.  If not, then you know you should look
at the server code more carefully.  If so, then the client code is probably
at fault.

Jean-Paul