[Twisted-Python] Questions about the very nice AMP protocol
exarkun at divmod.com
Fri Nov 17 08:32:26 EST 2006
On Thu, 16 Nov 2006 23:23:43 -0700, Brian Granger <ellisonbg.net at gmail.com> wrote:
>I currently use Perspective Broker for a number of projects. As time
>has gone by I have really come to appreciate having a full two-way
>network protocol. But, my company has lots of Java programmers and
>they do lots of "serious" (read, pain in the ass) web services and
>grid services stuff. I would like to be able to get the many Python
>things I have playing nicely with the many Java things floating around
>Thus, AMP is extremely attractive. There is one problem that I have
>though. We do high performance scientific computing and deal with
>extremely large tera/peta-byte data sets.
>Thus we need network protocols that can send large amounts of data
>around. The focus of AMP
>of small messages thus presents a problem. There are really two usage
>cases that I have in mind:
>1. Sending larger (maybe 100's of Mb) objects around that do fit in
>memory. These can be serialized easily (w/o creating a big pickle),
>but I need to make sure that Twisted doesn't make extra copies of them
>during the transfer.
>2. Sending even bigger things that don't fit into memory.
>Any thoughts on the best way to address these questions using AMP.
>Here are my thoughts:
>1. Use a multi-connection approach like FTP does. Use AMP for
>control and the other connection
>for the binary data. It would be easy to use producers/consumers in
>this channel to handle the large data problems above. I don't like
>this because I often need to ssh tunnel the protocol through firewalls
>- two connections is unpleasant.
>2. Use AMP's inner protocol to run two protocols simultaneously. My
>understanding is that AMP doesn't support switching back and forth
>between AMP and its inner protocol. Would it be crazy to try this
>3. Try to modify AMP itself to handle the large objects itself by
>registering Producers with the underlying transport.
This is the planned direction for AMP, if that makes any difference to
>It may sound like I just want something like FTP, but I also need to
>send lots application specific control messages as well - and these
>really need to be two way.
>Any thoughts would be greatly appreciated.
More information about the Twisted-Python