[Twisted-Python] dataReveived() buffer best practice?
glyph at twistedmatrix.com
Sat Oct 8 19:56:05 EDT 2011
On Oct 7, 2011, at 6:49 PM, Phil Mayers wrote:
> On 10/07/2011 02:11 PM, Glyph wrote:
>>> It is quite unusual (though not unheard of) for something to
>>> re-segment the TCP stream. IP level fragmentation might occur, but
>>> it's relatively uncommon in todays IP networks, and is anyway
>>> irrelevant to TCP - the TCP stack will only see a reassembled IP
>> I am skeptical; my recollection (the last time I worked at this layer
>> of the network) is that this happens all the time over the public
>> internet between diverse endpoints.
> Are we talking about TCP re-segmentation, or IP fragmentation?
Pretty sure I'm talking about TCP re-segmentation.
> If you mean TCP re-segmentation, then I have no data. The only places
> I've personally seen it are in TCP terminating load-balancers (F5,
> etc.), NAT64 devices, and as a side effect in higher-layer proxies
> (Squid, etc.).
Hrm. Now that I think of it, the packets I was examining were probably coming from environments that were heavily managed by "smart" networking hardware, although I could only see the edges of the network. So it was probably a combination of high-level transparent proxies and wacky load-balancing and caching hardware.
> If you mean IP fragmentation, I claim that fragmentation by routers in
> the internet is rare. Whether it's 1% or 0.001% I couldn't say, but the
> vast majority of the time, routers do not end up fragmenting. For one
> thing, an awful lot of OSes send TCP segments with "don't fragment" set,
> to trigger ICMP error messages and path MTU discovery. For another, a
> lot of low MTU networks employ TCP MSS clamping to prevent this ever
That's interesting, and good to know :).
> For what it's worth, this "layer of the network" is my day job, but I
> don't claim oracle status.
>> But the sizes are also not important. The point is that TCP is about
>> streams, not packets, and you have to deal with arbitrary chunking if
>> you want your code to work right. What layer of the network this
>> happens at is not important to your code :).
Since this is your day job - is there a formally-correct name for this phenomenon? I usually refer to it as "packet fragmentation" even though I know that's not really it - as you say, IP fragmentation doesn't really happen. It's not like it's really TCP re-segmenting either, because one send() is not necessarily one segment.
Since it's such an important concept that every Twisted developer needs to know, I'd love to have a good word for it :).
More information about the Twisted-Python