[Twisted-Python] dataReveived() buffer best practice?

[Phil Mayers]

On 10/07/2011 02:11 PM, Glyph wrote:

>> It is quite unusual (though not unheard of) for something to
>> re-segment the TCP stream. IP level fragmentation might occur, but
>> it's relatively uncommon in todays IP networks, and is anyway
>> irrelevant to TCP - the TCP stack will only see a reassembled IP
>> packet.
>
> I am skeptical; my recollection (the last time I worked at this layer
> of the network) is that this happens all the time over the public
> internet between diverse endpoints.

Are we talking about TCP re-segmentation, or IP fragmentation?

If you mean TCP re-segmentation, then I have no data. The only places 
I've personally seen it are in TCP terminating load-balancers (F5, 
etc.), NAT64 devices, and as a side effect in higher-layer proxies 
(Squid, etc.).

If you mean IP fragmentation, I claim that fragmentation by routers in 
the internet is rare. Whether it's 1% or 0.001% I couldn't say, but the 
vast majority of the time, routers do not end up fragmenting. For one 
thing, an awful lot of OSes send TCP segments with "don't fragment" set, 
to trigger ICMP error messages and path MTU discovery. For another, a 
lot of low MTU networks employ TCP MSS clamping to prevent this ever 
happening.

For what it's worth, this "layer of the network" is my day job, but I 
don't claim oracle status.

> But the sizes are also not important.  The point is that TCP is about
> streams, not packets, and you have to deal with arbitrary chunking if
> you want your code to work right.  What layer of the network this
> happens at is not important to your code :).

Agreed.