[Twisted-Python] Sending longer messages in AMP

Gavin Panella gavin at gromper.net
Fri Nov 14 04:59:02 MST 2014


On 13 November 2014 18:19,  <exarkun at twistedmatrix.com> wrote:
> On 02:57 pm, gavin at gromper.net wrote:
>>
>> Hi,
>>
>> We're using AMP and are starting to hit TooLong errors when scaling
>> our application. In one respect it's a sign that we should do
>> something like paging large requests and responses, but that's a lot
>> more work, and comes with its own problems. We also don't need
>> particularly large payloads: right now, a limit of ~500kiB would
>> allow us to scale as far as we need and beyond.
>>
>> I've put together a fork of Twisted's AMP implementation that uses
>> 32-bit length prefixes everywhere, though it limits the maximum
>> message size to 2MiB. Every other aspect of it is the same so it's a
>> drop-in replacement, as long as both ends of a connection use it.
>> However, there's no negotiation phase so it's completely incompatible
>> on the wire. The overhead of a few extra bytes is negligible for our
>> use cases, where the networks are all assumed to be low-latency
>> high-bandwidth LANs.
>>
>> Are there any reasons that we shouldn't be doing this? Was there a
>> good reason for 16-bit length prefixes that still holds? Should we be
>> doing something else?
>
>
> The short length limit is in place to encourage two things:
>
>  * messages that can be processed in a
>    cooperative-multitasking-friendly way
>
>  * the AMP channel can reliably used to multiplex multiple operations
>
> The limit encourages the former by limiting the total amount of data
> it's possible to receive in a single command. Of course, you can still
> do ridiculously complicated work based on a small bit of data so this
> doesn't guarantee that no matter what you do you'll be safe. But doing
> even something simple on a ridiculously large amount of data is
> probably guaranteed to take a while.
>
> The limit encourages the latter by putting a limit on the data that
> needs to be transferred to complete any one command (or answer).
> Again, this isn't a guarantee of safety (you could always have a `for
> i in range(1e10): callRemote(...)` loop and clog up the channel for
> ages) but it pushes things a bit more in that direction.
>
> At ClusterHQ we *also* maintained a fork of AMP with this limit
> raised. Basically, it worked. It did let us get into the kind of
> trouble that the limit was supposed to try to avoid (in particular it
> let us send around messages that would take longer and longer to be
> processed - in a system where keeping latency down was actually sort
> of important; fortunately we had *worse* problems introducing latency
> so this in particular never bit us too hard ;).
>>
>> If I assume that the answers are all no, would someone find this
>> protocol useful if we submitted it for inclusion in Twisted itself?
>
> There are better solutions to the problem. The trouble is that they're
> also more work to implement. ;) I think Twisted should hold out for
> the better solutions though, not adopt a like-AMP-but-with-different-
> hard-coded-limits solution.
>
> What are the better solutions? Library support for paging, basically.
> Or, to consider things more generally, library support for streaming.
> The AMP implementation in Twisted (note, not the *protocol*) should be
> extended to make it easy to pass arbitrarily large streams of data
> around - suitably broken into smaller pieces at the box level.
>
> As of right now, the way I'd do that is by introducing a new argument
> type (or two) supporting `IProducer` and `IConsumer`. Pass in an
> `IProducer` and the library will take the necessary steps to read data
> out of it, chunk it up into <=16kB chunks, and re-assemble them on the
> receiving side (as another `IProducer`).
>
> There are two reasons I'm not working on this right now (apart from
> the standard reasons of not having time to do so ;):
>
>  1) IProducer / IConsumer aren't amenable to this kind of decoupling.
> You can register a producer with a consumer but you can't register a
> consumer with a producer. By the time you give the IProducer to AMP,
> it's too late to tell it you want it to send its data into the AMP
> implementation for the necessary handling. We worked around this in
> twisted.web.client.Agent by introducing a new IProducer-like
> interface. It solves the basic problem but it doesn't go any further
> to improve the usability of the interfaces.
>
>  2) Tubes. Glyph is working on a replacement for IProducer/IConsumer
> that does go a lot further to improve usability. With this promise of
> a bright, prosperous future looming, it's hard to get excited about
> implementing for AMP a just-barely-good-enough solution like the one
> used by Agent (in particular, with the knowledge that the tubes
> solution will be API incompatible and we'll most likely want to
> deprecate the IProducer/IConsumer thing).

Thank you for such a comprehensive reply. It's really helped.

As a result we're going to stick with the 16-bit AMP, by changing the
calls we're making. In one particular case we'll change our code to
regularly (with a throttle) fetch small batches from a priority queue
instead of getting all-the-things periodically. That'll avoid the need
to switch to 32-bit AMP, and will also be good for the responsiveness of
the overall application.

The Tubes stuff is interesting. As that matures we may look at doing the
work to bring Tubes and AMP together.

Gavin.




More information about the Twisted-Python mailing list