Opened 7 years ago

Closed 7 years ago

Last modified 7 years ago

#5094 enhancement closed duplicate (duplicate)

FileDescriptor.SEND_LIMIT could be much higher

Reported by: Itamar Turner-Trauring Owned by:
Priority: normal Milestone:
Component: core Keywords:
Cc: Branch:
Author:

Description

I see no particular reason why we should only send 128KB at a time; it just slows down transmission when large amounts of data are being sent... assuming the OS will accept more (see below). I would say just try to write everything in the buffer, or if this will increase potential memory usage too much increase to 1MB or something.

At one point James Knight documented that:

"In Unix, socket.send(buf) will buffer as much of buf as it has space for, and then return how much it accepted. This could be 0 or up to something around 128K. If you send some data and then some more, it will append to the previous buffer.

In Windows, socket.send(buf) will either accept the entire buffer or raise ENOBUFS. Testing indicates that it will internally buffer any amount up to 50MB (this seems to be the total for either the process or the OS, I'm not sure). However, it will not incrementally accept more data to append to a socket's buffer until the big buffer has been completely emptied (seemingly down to the SO_SNDBUF length, which is 8192), but rather raises WSAEWOULDBLOCK instead."

So, on Windows this may be bad idea. On Unix, I'm not sure how he came up with 128K; possibly it was the default SO_SNDBUF size on some platform. On my Ubuntu box SO_SNDBUF seems to default to 16KB. If SO_SNDBUF is in fact what the OS uses to determine how much to write in one go (but who knows, maybe it's some buffer in the ethernet card or something?), perhaps we should also expose an API for changing it and SO_RCVBUF.

Change History (5)

comment:1 Changed 7 years ago by Itamar Turner-Trauring

Some simple tests on Linux 2.6.38 (Ubuntu 11.04) suggests that over loopback at least, a socket will often write more than 128KB, e.g. I saw 212992 bytes written a time.

comment:2 Changed 7 years ago by Glyph

I think this may be a duplicate of #4089.

comment:3 Changed 7 years ago by Itamar Turner-Trauring

Resolution: duplicate
Status: newclosed

Just to clarify, I wrote 212992 even though SO_SNDBUF was only 16384... so you can't rely on SO_SNDBUF to set the SEND_LIMIT.

And yes, duplicate.

comment:4 in reply to:  3 Changed 7 years ago by Glyph

Replying to itamar:

Just to clarify, I wrote 212992 even though SO_SNDBUF was only 16384... so you can't rely on SO_SNDBUF to set the SEND_LIMIT.

And yes, duplicate.

Can you attach the test program that you used to verify this so that we can run it on multiple platforms and collect some experimental output?

comment:5 Changed 7 years ago by Glyph

Oops wrong ticket.

Note: See TracTickets for help on using tickets.