[Twisted-Python] Sending longer messages in AMP

Gavin Panella gavin at gromper.net
Thu Nov 13 07:57:40 MST 2014


Hi,

We're using AMP and are starting to hit TooLong errors when scaling
our application. In one respect it's a sign that we should do
something like paging large requests and responses, but that's a lot
more work, and comes with its own problems. We also don't need
particularly large payloads: right now, a limit of ~500kiB would allow
us to scale as far as we need and beyond.

I've put together a fork of Twisted's AMP implementation that uses
32-bit length prefixes everywhere, though it limits the maximum
message size to 2MiB. Every other aspect of it is the same so it's a
drop-in replacement, as long as both ends of a connection use it.
However, there's no negotiation phase so it's completely incompatible
on the wire. The overhead of a few extra bytes is negligible for our
use cases, where the networks are all assumed to be low-latency
high-bandwidth LANs.

Are there any reasons that we shouldn't be doing this? Was there a
good reason for 16-bit length prefixes that still holds? Should we be
doing something else?

If I assume that the answers are all no, would someone find this
protocol useful if we submitted it for inclusion in Twisted itself?
The code right now is a straight copy of amp.py and test_amp.py with
changes to 32-bit length prefixes everywhere, but for upstreaming we'd
probably propose instead to modify the original to have an optional
negotiation phase, and to make the maximum message size a parameter.

Thanks!

Gavin.



More information about the Twisted-Python mailing list