[Twisted-Python] AMP message length limit

[Glyph Lefkowitz]

> On Nov 22, 2015, at 17:51, Oon-Ee Ng <ngoonee.talk at gmail.com> wrote:
> 
> I've just (to my surprise) hit this. As I understand from searching
> around, AMP messages are limited to ~64k due to the length prefix
> being 16-bit. A change in my internal data being sent (using dicts
> rather than lists) kicked one of my messages to way over that limit.

I'm sorry that this was an unpleasant surprise.  I wish that we had a better way of getting this across up-front :-).  However, it seems like the length limit is doing its job in terms of constraining your protocol design to not have individual messages "hog" the wire...

> There's a bit of discussion here -
> http://twistedmatrix.com/pipermail/twisted-python/2014-November/028947.html
> 
> Is there an internal twisted solution planned, or should I go ahead
> and roll my own paging code? If the latter (as I strongly suspect),
> could I get some comments on this idea:-

Definitely the latter if you have a short time frame.  How big are your messages?  If your limit is still fairly small (5M, let's say) but much bigger than 64k there are other options you can use.

> Original amp.Command had a single argument (amp.ListOf(amp.String())
> and no response

> Modified amp.Command, 4 arguments and 1 response
> ID (sequentially generated by producer) - amp.Integer()
> CurPage - amp.Integer()
> TotalPage - amp.Integer()
> ActualData - amp.ListOf(amp.String())
> Response - RecievedPage - amp.Integer()

Implementing a paging API like this is exactly what the length limit is designed to encourage you to do - it is much more flexible, since you can request a subset of pages, and continue receiving things other than pages while the data is being streamed to you.

> Questions:-
> 1. ID is so the client can be sure not to concatenate different lists

This... is correct, but doesn't sound like a question.  Is it meant to be?

> 2. Do I need a response at all?

No.  You can tell AMP not to bother generating the protocol-level response by setting the requiresAnswer flag on your Command to False: <https://twistedmatrix.com/documents/15.4.0/api/twisted.protocols.amp.Command.html#requiresAnswer>

> 3. Should I attempt to plug as many list items as possible into each
> page (requires length checking of json-encoded strings and repeated
> encoding/checks) or just choose a suitable limit of list items (my
> current max length is about 200 characters and average is 71) of maybe
> 300 list items per message? My current list is about 1k items in all,
> and it's only going to get bigger.

Why are you encoding as _both_ JSON and AMP?

I'd say you should do the length-checking, because you still might end up with list items that are larger than expected if they're variable size.

> 4. I was intrigued by the mention of 'Tubes' in the link above. Found
> it here - https://tubes.readthedocs.org/en/latest/tube.html - should I
> be using that instead? I'm writing a homegrown app which will only
> really need (at this point) to communicate with itself and copies of
> itself, and settled with AMP as being a simple way of achieving that.

I would love it if you would help me test out and develop Tubes.  If it is a small homegrown app it might be a good use-case.  There are pros and cons: Tubes has higher test coverage and cleaner code since it was developed much more recently; but, it still has very limited functionality, badly broken areas, and no compatibility guarantees, because it's still somewhat experimental.

However, Tubes is a way of implementing protocols, whereas AMP is an implementation of a request/response protocol.  If you use Tubes, you'll need to do an implementation of AMP (or something like it) in order to issue requests and give responses.  If I were you, especially since you've already figured out paging, I would probably just stick with AMP and Twisted as-is.

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20151122/4e1f2ce9/attachment-0002.html>