[Twisted-Python] pushing out same message on 100k TCPs

[Glyph Lefkowitz]

On Feb 10, 2012, at 12:49 PM, Phil Mayers wrote:

> On 10/02/12 16:56, Tobias Oberstein wrote:
>> Hi there,
>> 
>> what is the most efficient/performant way of doing the following?
>> 
>> I have a short message prepared .. say a string of 100 octets.
>> I want to push out that _same_ string on 100k connected TCPs (on a server).
>> 
>> ==
>> 
>> My thinking was: ideally, the 100 bytes would be transferred to kernel/NIC space
>> just _once_, and then the kernel is only told to resend that buffer on the 100k
>> connected TCPs.
>> 
>> Does that make sense, is that even possible, with Twisted, or in general?
> 
> Not really, no.
> 
> The problem is that TCP requires the sender of data to buffer so that it 
> can re-send. The only way to store one copy of the data whilst doing 
> this would be to store the socket buffer as a (fairly complex) linked 
> list of reference-counted blocks, and use scatter-gather IO to the 
> network card.
> 
> So the kernel would have to copy the data 100k times anyway, to store it 
> in the per-socket buffer until it was ACKed, or maintain a large and 
> complex data structure so that it could use one copy.
> 
> Therefore, by moving the work to the kernel, all you've done is consume 
> valuable kernel memory, in return for saving the syscall overhead. 
> Classic space/time tradeoff.
> 
> If you were using UDP, then in theory this might be possible, but there 
> are no APIs that I know of, except for multicast (where you only send 
> one copy of the data, and the network duplicates it).
> 
> In short; this kind of thing seems easy and desirable but actually it's 
> really hard and not useful.

Not to mention the fact that inevitably, you probably are going to want some security on those connections, which means TLS, which means individual crypto connections.

I believe the best model for this kind of high-volume reliable-multicast-over-unicast is a spanning tree, like what IRC uses for server-to-server communication.  If you have 100,000 concurrent, active connections, you're already probably beyond the CPU constraints of a single machine.  So have your clients connect (via round-robin DNS, or whatever other mechanism makes sense) to a variety of different servers, then have the servers connected to each other in a pattern such that one server tells ten friends, and they each tell ten friends, and so on, until each server is only responsible for spewing out data to a reasonable number of connections (5000 maybe?).

-glyph