[Twisted-Python] Compressing PB communication

Tzahi Fadida Tzahi.ML at gmail.com
Fri Jun 30 05:24:43 EDT 2006

On Friday 30 June 2006 03:44, glyph at divmod.com wrote:
> On Thu, 29 Jun 2006 22:01:22 +0300, Tzahi Fadida <tzahi.ml at gmail.com> wrote:
> >No, that is new to me. I cannot break it up.
> >can it be done using that writeSequence or Write in the ProtocolWrapper?
> >I don't want to handle this on the application/PB level.
> >I can see though that this is confined to the banana.py and cbanana,
> >i can change it there as a last resort, right?
> >640kb looks arbitrary to me, times is changing, what was considered
> >large in the past is small in the present. At the very least it should be
> >configurable.
> Not really.  PB is optimized (heavily optimized) for lots of small
> control-channel messages.  Serializing 20MB of data with Jelly would
> probably stop your process for a good 30 seconds, and due to the way that
> the original Jelly was designed, it cannot be processed incrementally; you
> will end up allocating something like 100MB of memory just to get the data
> serialized and the packet sent, if you raise the limit.

30 seconds of doing what? serializing?
Let me try something else, what if i want to replace jelly with pickle for
a special Server and Client factory for sending messages from the Server to
the client only. If the client sends a message to the server, they must
be jellied. The idea is that you are consciously saying that the client 
completely trust the server.
I think this is a good idea for some security model that also want performance 
and resource savings. 

> Feel free to do so; if multi-second messages and multi-hundred-megabyte
> memory costs are acceptable to you then you can just change the arbitrary
> limit.  I won't accept patches to make it easily configurable in Twisted
> though; IMHO that is just bad design.  You should instead be factoring your
> data to be produced on demand in a series of messages rather than one giant
> message.  In addition to lower overall memory cost and more immediate
> feedback, that has the additional advantage of providing you with a way to
> interleave other messages on the same channel.

I don't understand, you are saying that twisted does not send portions, i.e.
it is blocking? that doesn't sound right to me, even if i send 20mb of data
it should be portioned and let other transfer also get the chance.
I was under the impression that twisted is multiplexing even on the channel 
level. I can, of course, always open 2 channels but this is evil.

> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Tzahi Fadida
Blog: http://tzahi.blogsite.org | Home Site: http://tzahi.webhop.info

More information about the Twisted-Python mailing list