[Twisted-Python] Sending large amounts of data over Pb
exarkun at divmod.com
Tue Apr 25 17:08:53 EDT 2006
On Tue, 25 Apr 2006 11:32:21 -0600, Gabe Rudy <rudy at goldenhelix.com> wrote:
>In a few specific cases (not as a general case) I'm sending a significant list
>of items over pb in a pb.RemoteCache/pb.Cacheable object pair. To give you an
>idea, it's a list of ~4000 tuples where each tuples has about ~10 strings of
>Right now twisted works fine, but takes about 10 seconds to
>serialize/send/unserialize the data, or I guess in twisted terms,
>banana/send/recv/banana the data.
>Since my list is all primitives I thought maybe pickling it before sending
>would speed it up as banana would have to do less work, but I get a
>twisted.spread.banana.BananaError: string is too long to send (706378)
>Is that an arbitrary length limit or technically significant?
>Is there other methods of speeding this up?
>Or alternatively, is it possible (within pb) to receive the state of a
>RemoteCache or RemoteObject in definable chunks (say 100 items from the list
>at a time), allowing me to start displaying the data in my UI and adding more
>to it as it comes async?
I'm not sure how you are using RemoteCache, but the general technique here is to use a pager. Take a look at twisted.spread.util.Pager, and the two examples/tools that come with it, StringPager and FilePager.
More information about the Twisted-Python