[Twisted-Python] Multicast XMLRPC

glyph at divmod.com glyph at divmod.com
Sat Aug 26 15:43:28 MDT 2006


On Sat, 26 Aug 2006 10:14:48 -0400, "Chaz." <eprparadocs at gmail.com> wrote:

>Right now I am trying to find a solution to an interesting problem: how to 
>find a file without knowing exactly where it exists in the network. You have 
>to do this to make the system scale nicely.

>Basically each node holds information about the files (aka objects) it 
>stores. I do this so that I don't have a central database any where (this 
>allows the system to scale differently. With a central database I would have 
>that set of servers scale differently than the storage nodes).

>Now I can build a set of machines that are the distributed database machines 
>- each storing something - and querying them for where the file lives; this 
>would narrow the machines I have to directly talk to, but it feels wrong. 
>This is sort of a variation of the hub-and-spoke that Glyph talked about. 
>But having said that I am trying to determine if I can get away from that 
>and just go to a very unstructured environment (without intermediate 
>database nodes).

This sounds an awful lot like a distributed hashtable.  It does implicitly use an overlay network, but not a hub-and-spoke overlay network.

I'm not intimately familiar with the algorithms involved, so rather than try to describe them, I'll just refer you to the relatively nice wikipedia page on the topic:

    http://en.wikipedia.org/wiki/Distributed_hash_table

There is also a project in Python (not Twisted though) which may serve as an example:

    http://thecircle.org.au/

Are these ideas useful?  Have you looked at them before?




More information about the Twisted-Python mailing list