[Twisted-Python] AMP message length limit

Oon-Ee Ng ngoonee.talk at gmail.com
Sun Nov 22 15:51:57 MST 2015


I've just (to my surprise) hit this. As I understand from searching
around, AMP messages are limited to ~64k due to the length prefix
being 16-bit. A change in my internal data being sent (using dicts
rather than lists) kicked one of my messages to way over that limit.

There's a bit of discussion here -
http://twistedmatrix.com/pipermail/twisted-python/2014-November/028947.html

Is there an internal twisted solution planned, or should I go ahead
and roll my own paging code? If the latter (as I strongly suspect),
could I get some comments on this idea:-

Original amp.Command had a single argument (amp.ListOf(amp.String())
and no response

Modified amp.Command, 4 arguments and 1 response
ID (sequentially generated by producer) - amp.Integer()
CurPage - amp.Integer()
TotalPage - amp.Integer()
ActualData - amp.ListOf(amp.String())
Response - RecievedPage - amp.Integer()

Questions:-
1. ID is so the client can be sure not to concatenate different lists
2. Do I need a response at all?
3. Should I attempt to plug as many list items as possible into each
page (requires length checking of json-encoded strings and repeated
encoding/checks) or just choose a suitable limit of list items (my
current max length is about 200 characters and average is 71) of maybe
300 list items per message? My current list is about 1k items in all,
and it's only going to get bigger.
4. I was intrigued by the mention of 'Tubes' in the link above. Found
it here - https://tubes.readthedocs.org/en/latest/tube.html - should I
be using that instead? I'm writing a homegrown app which will only
really need (at this point) to communicate with itself and copies of
itself, and settled with AMP as being a simple way of achieving that.

Thanks for the time.



More information about the Twisted-Python mailing list