[Twisted-Python] Multicast XMLRPC
glyph at divmod.com
glyph at divmod.com
Fri Aug 25 13:54:42 EDT 2006
On Fri, 25 Aug 2006 13:07:49 -0400, "Chaz." <eprparadocs at gmail.com> wrote:
>I will state what I thought was obvious: I need to make "calls" to thousands
>of machines to do something. I want to minimize the overhead both of making
>the call and the machines sending back the responses.
Maybe you could start off a little bit further back into the problem today. Like, "I got up this morning, and I thought, 'I would like some toast.', but I don't know how to make toast, so I wanted to design a 100,000 node parallel neural network to develop a receipie for toast."
Perhaps then someone on this list could relate their toast development experiences, such as "using a TCP-based tree topology similar to IRC servers has been sufficient in my experience for toast-oriented data exchange although I have been using a parallelized coordinated genetic algorithm rather than a neural network to develop an optimal crunch/warmth experience", or possibly "ToastVortex, my Twisted-basted toast application server is available at http://toastvortex.example.com/" or better yet, "buy a toaster and put some bread in it".
>TCP is pretty resource intensive so I need something else. I think a
>reliable datagram service on top of some underlying transport is the way to
>go (on top of multicast/broadcast/IP is what I am thinking about).
TCP's "resource" consumption is localized in an a highly optimized environment; in OS kernels, where the TCP stack is tuned constantly by thousands of people, in routing hardware that is specialized to give TCP traffic priority to improve performance, and in the guts of the public internet that runs such hardware and is constantly monitored and tweaked to give TCP even more of a boost. Any custom multicast protocol you develop, while perhaps theoretically better than TCP, is possibly going to get swamped by the marginalia that TCP has spent decades eradicating. In Python, you're going to be doing a lot of additional CPU work. For example, TCP acks often won't even be promoted to userspace, whereas you're going to need to process every unicast acknowledgement to your multicast message separately in userspace.
While my toast network deployments are minimal, I *have* written quite a few multi-unicast servers, some of which processed quite a high volume of traffic acceptably, and in at least one case this work was later optimized by another developer who spent months working on a multicast replacement. That replacement which was later abandoned because the deployment burden of a large-scale multicast-capable network was huge. That's to say nothing of the months of additional time required to develop and properly *test* such a beast.
You haven't said what resources TCP is consuming which are unacceptble, however, Is it taking too much system time? Too much local bandwidth? Is your ethernet experiencing too many collisions? Are you concerned about the cost of public internet bandwidth overages with your service provider? What's your network topology? It would be hard to list the answers to all of these questions (or even exhaustively ask all the questions one would need to comment usefully) but one might at least make guesses that did not fall too wide of the mark if one knew what the application in question were actually *doing*.
In any event, XML-RPC is hardly a protocol which is famous for its low resource consumption on any of these axes, so if you're driven by efficiency concerns, it seems an odd choice to layer on top of a hand-tuned multicast-request/unicast-response protocol.
More information about the Twisted-Python