[Twisted-Python] pushing out same message on 100k TCPs
p.mayers at imperial.ac.uk
Sat Feb 11 05:48:45 EST 2012
On 02/10/2012 09:54 PM, Glyph Lefkowitz wrote:
> I believe the best model for this kind of high-volume
> reliable-multicast-over-unicast is a spanning tree, like what IRC
For what it's worth, real IP multicast is quite commonly used for
distributing short messages to many clients in realtime in some closed
networks, such as financial trading systems. With good network equipment
that can handle low- or zero-loss timely delivery, it does work very well.
Over the internet - less well, and even if your target networks are
multicast-enabled, something like NORM or FEC is required to fill in the
gaps, which inevitably adds latency.
Add to which the fact that you want your protocol to be
congestion-control compatible, which is seriously challenging with real
multicast, and if you want crypto you've got to have some kind of group
> uses for server-to-server communication. If you have 100,000
> concurrent, active connections, you're already probably beyond the
> CPU constraints of a single machine. So have your clients connect
> (via round-robin DNS, or whatever other mechanism makes sense) to a
> variety of different servers, then have the servers connected to each
> other in a pattern such that one server tells ten friends, and they
> each tell ten friends, and so on, until each server is only
> responsible for spewing out data to a reasonable number of
> connections (5000 maybe?).
I agree, this is probably the best model at the moment. It has the
advantage that if you need to scale globally, you can push the "caching"
nodes closer to the clients, and use geographical-aware routing to push
clients onto closer nodes.
More information about the Twisted-Python