[Twisted-Python] Sending longer messages in AMP

[exarkun at twistedmatrix.com]

On 02:57 pm, gavin at gromper.net wrote:
>Hi,
>
>We're using AMP and are starting to hit TooLong errors when scaling
>our application. In one respect it's a sign that we should do
>something like paging large requests and responses, but that's a lot
>more work, and comes with its own problems. We also don't need
>particularly large payloads: right now, a limit of ~500kiB would allow
>us to scale as far as we need and beyond.
>
>I've put together a fork of Twisted's AMP implementation that uses
>32-bit length prefixes everywhere, though it limits the maximum
>message size to 2MiB. Every other aspect of it is the same so it's a
>drop-in replacement, as long as both ends of a connection use it.
>However, there's no negotiation phase so it's completely incompatible
>on the wire. The overhead of a few extra bytes is negligible for our
>use cases, where the networks are all assumed to be low-latency
>high-bandwidth LANs.
>
>Are there any reasons that we shouldn't be doing this? Was there a
>good reason for 16-bit length prefixes that still holds? Should we be
>doing something else?

The short length limit is in place to encourage two things:

  * messages that can be processed in a cooperative-multitasking-friendly 
way

  * the AMP channel can reliably used to multiplex multiple operations

The limit encourages the former by limiting the total amount of data 
it's possible to receive in a single command.  Of course, you can still 
do ridiculously complicated work based on a small bit of data so this 
doesn't guarantee that no matter what you do you'll be safe.  But doing 
even something simple on a ridiculously large amount of data is probably 
guaranteed to take a while.

The limit encourages the latter by putting a limit on the data that 
needs to be transferred to complete any one command (or answer).  Again, 
this isn't a guarantee of safety (you could always have a `for i in 
range(1e10): callRemote(...)` loop and clog up the channel for ages) but 
it pushes things a bit more in that direction.

At ClusterHQ we *also* maintained a fork of AMP with this limit raised. 
Basically, it worked.  It did let us get into the kind of trouble that 
the limit was supposed to try to avoid (in particular it let us send 
around messages that would take longer and longer to be processed - in a 
system where keeping latency down was actually sort of important; 
fortunately we had *worse* problems introducing latency so this in 
particular never bit us too hard ;).
>If I assume that the answers are all no, would someone find this
>protocol useful if we submitted it for inclusion in Twisted itself?

There are better solutions to the problem.  The trouble is that they're 
also more work to implement. ;)  I think Twisted should hold out for the 
better solutions though, not adopt a like-AMP-but-with-different-hard- 
coded-limits solution.

What are the better solutions?  Library support for paging, basically. 
Or, to consider things more generally, library support for streaming. 
The AMP implementation in Twisted (note, not the *protocol*) should be 
extended to make it easy to pass arbitrarily large streams of data 
around - suitably broken into smaller pieces at the box level.

As of right now, the way I'd do that is by introducing a new argument 
type (or two) supporting `IProducer` and `IConsumer`.  Pass in an 
`IProducer` and the library will take the necessary steps to read data 
out of it, chunk it up into <=16kB chunks, and re-assemble them on the 
receiving side (as another `IProducer`).

There are two reasons I'm not working on this right now (apart from the 
standard reasons of not having time to do so ;):

  1) IProducer / IConsumer aren't amenable to this kind of decoupling. 
You can register a producer with a consumer but you can't register a 
consumer with a producer.  By the time you give the IProducer to AMP, 
it's too late to tell it you want it to send its data into the AMP 
implementation for the necessary handling.  We worked around this in 
twisted.web.client.Agent by introducing a new IProducer-like interface. 
It solves the basic problem but it doesn't go any further to improve the 
usability of the interfaces.

  2) Tubes.  Glyph is working on a replacement for IProducer/IConsumer 
that does go a lot further to improve usability.  With this promise of a 
bright, prosperous future looming, it's hard to get excited about 
implementing for AMP a just-barely-good-enough solution like the one 
used by Agent (in particular, with the knowledge that the tubes solution 
will be API incompatible and we'll most likely want to deprecate the 
IProducer/IConsumer thing).

Jean-Paul
>The code right now is a straight copy of amp.py and test_amp.py with
>changes to 32-bit length prefixes everywhere, but for upstreaming we'd
>probably propose instead to modify the original to have an optional
>negotiation phase, and to make the maximum message size a parameter.
>
>Thanks!
>
>Gavin.
>
>_______________________________________________
>Twisted-Python mailing list
>Twisted-Python at twistedmatrix.com
>http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python