[Twisted-Python] Streaming in AMP

[Jamu Kakar]

Hi,

Glyph has been kind enough to share some of his ideas about adding
streaming support to AMP via private mail.  He suggested that the
discussion should really be on the list.  Here's the relevant technical
details we've discussed so far:

(I made the comments in quotes; Glyph's response follows)

>When you get a chance to get to the AMP streaming spec please let me
>know.  I'm interested in learning more about AMP and possibly helping
>out with it.

Since there is basically about a 5% probability that I will actually
manage to write something that looks like a spec, let me take this
random email response as an opportunity to get started.

The protocol limitation in AMP of 64k per value is intentional, and in
fact, should regarded as a very high end for a per-packet payload.  The
idea of AMP is that the connection should be a multiplexable resource
which can do lots of things at the same time; if you start plugging it
up with megabyte-at-a-time payloads, then when you send a packet over
it, it's not going to get there in a reasonable time frame.

So the idea behind AMP streaming is that there should be a parameter
type which presents an API that allows you to deal with a "large" value
that might be a multi-gigabyte stream, chunked up into smaller fragments
so that you can have one connection downloading multiple things.

I haven't completely decided on the API yet, but just to give you an
idea of where I'm going:

    class Put(amp.Command):
        arguments = [('upload', amp.Upload()),
                     ('filename', amp.Unicode())]

    # ...
    def put(upload, filename):
        def saveResult(result):
            f = file(filename, 'wb')
            f.write(result)
            f.close()
        upload.fullResultAsString().addCallback(saveResult)
        return {}
    Put.responder(put)

This is a straw-man API because the *suggested* API for doing something
like this would be to incrementally write the results to a file as
they're available, but a hacky convenience API like this might be useful
in some applications (and as an example it's less code to write).
Another, possibly better convenience API might be
'fullResultAsTemporaryFile'.

The implementation technique here would be to make the AMP protocol
object a multiplexable consumer, and establish a convention (perhaps
some built-in commands?) for sending chunks of a previously-identified
file object.  You can find the notifications involved in this process
documented here:

http://twistedmatrix.com/projects/core/documentation/howto/producers.html

Is this helpful as the beginnings of a specification?  Let me know if
you need more information.