[Twisted-Python] Sending large amounts of data over Pb
rudy at goldenhelix.com
Tue Apr 25 13:32:21 EDT 2006
In a few specific cases (not as a general case) I'm sending a significant list
of items over pb in a pb.RemoteCache/pb.Cacheable object pair. To give you an
idea, it's a list of ~4000 tuples where each tuples has about ~10 strings of
Right now twisted works fine, but takes about 10 seconds to
serialize/send/unserialize the data, or I guess in twisted terms,
banana/send/recv/banana the data.
Since my list is all primitives I thought maybe pickling it before sending
would speed it up as banana would have to do less work, but I get a
twisted.spread.banana.BananaError: string is too long to send (706378)
Is that an arbitrary length limit or technically significant?
Is there other methods of speeding this up?
Or alternatively, is it possible (within pb) to receive the state of a
RemoteCache or RemoteObject in definable chunks (say 100 items from the list
at a time), allowing me to start displaying the data in my UI and adding more
to it as it comes async?
More information about the Twisted-Python