[Twisted-Python] Compressing PB communication

[glyph at divmod.com]

On Thu, 29 Jun 2006 22:01:22 +0300, Tzahi Fadida <tzahi.ml at gmail.com> wrote:

>No, that is new to me. I cannot break it up.
>can it be done using that writeSequence or Write in the ProtocolWrapper?
>I don't want to handle this on the application/PB level.
>I can see though that this is confined to the banana.py and cbanana,
>i can change it there as a last resort, right?
>640kb looks arbitrary to me, times is changing, what was considered
>large in the past is small in the present. At the very least it should be
>configurable.

Not really.  PB is optimized (heavily optimized) for lots of small control-channel messages.  Serializing 20MB of data with Jelly would probably stop your process for a good 30 seconds, and due to the way that the original Jelly was designed, it cannot be processed incrementally; you will end up allocating something like 100MB of memory just to get the data serialized and the packet sent, if you raise the limit.

Feel free to do so; if multi-second messages and multi-hundred-megabyte memory costs are acceptable to you then you can just change the arbitrary limit.  I won't accept patches to make it easily configurable in Twisted though; IMHO that is just bad design.  You should instead be factoring your data to be produced on demand in a series of messages rather than one giant message.  In addition to lower overall memory cost and more immediate feedback, that has the additional advantage of providing you with a way to interleave other messages on the same channel.