[Twisted-Python] Using pb/jelly for large objects

Jean-Paul Calderone exarkun at divmod.com
Wed Nov 9 08:58:40 EST 2005


On Wed, 9 Nov 2005 10:54:29 +0000, william.lewis at uk.bnpparibas.com wrote:
>Hi,
>
>I am looking to build a system that runs multiple processes across a LAN
>of win32 machines. The processes will be initialised with a large amount
>of data at startup, followed by many more smaller chunks of data with the
>results being returned across the network. I would like to
>represent/transfer the data using your pb.Copyable and/or pb.Cacheable
>objects. My main question relates to the fact that pb uses jelly to
>serialise these objects. Particularly the comments at the top of the jelly
>module that emphasise the fact that jelly should have a lot more
>optimization in order to gain acceptable large-scale performance and that,
>compared to pickle, jelly is a more portable, human readable, less
>efficient mechanism.
>
>So, would you use pb/jelly for transferring large chunks of data (e.g.
>~10s MB) to multiple nodes on a LAN?
>If not, could pb be used with, for example, pickle instead of jelly?

Depends on the form of the data.  If it is represented locally as large strings, or lists of integers or floats, or some other simple structure such as that, I would probably use a PB control channel and a secondary data channel.  The secondary channel could speak a trivial protocol, such as int32strings, which would be much faster than any general object serialization mechanism.

Of course, this is just an optimization.  I'd probably start out passing the data inline, and only worry about this once I saw it was causing problems, though keeping this in mind so the APIs are amenable to the change.

>Also, I notice that the pb, jelly and flavors modules are all classified
>as semi-stable. Could you give an indication of what this means in
>practice?

PB hasn't changed in a manner that is not backwards compatible in years.  There is, of course, the looming juggernaut of newpb...

>
>Thank you for your time. I think you have built a very useful/powerful
>framework and am looking forward to using it.

Thanks :)  I hope you have fun building your application.

Jean-Paul




More information about the Twisted-Python mailing list