Include sendfile(2) support into defaultreactor and make twisted.web use it

[PenguinOfDoom]

I've done a quick hack to this effect. It seems that sendfile(2)-enhanced
twisted.web is significantly closer to Apache speeds on large (5MB) files.
Benchmarks will be posted once somebody figures out how to properly measure the
speed.

[glyph]

can you upload a patch, even if it's a nasty hack?

[spiv]

Where's the patch?

[exarkun]

Yea where's the patch?

[therve]

The branch is here: http://twistedmatrix.com/trac/browser/branches/pahan/sendfile/sandbox/pahan?rev=10492

The original sendfile wrapper is from here:  http://snakefarm.org/.

[therve]

For people interested, I've try to update PenguinOfDoom's work in sendfile-585, and it seems to work fine.

[jknight]

Not having looked at the patch yet, some test suggestions:

1) If the sendfile syscall is present, but fails, the code deals sanely (fallback to read/send, I suppose, or else some kind of notification?)

1b) what if it fails because the file is unreadable or has an I/O error of some kind?

2) SSL sockets don't support sendfile

3) it works, or at least, doesn't blow up on *BSD (which has a sendfile, with a different signature)

[exarkun]

From a quick skim of the branch, a couple thoughts:

• there should be a new interface including sendfile (if having a sendfile method is the right way to expose this). hasattr/getattr calls like the one in static.py result in bugs
• fallback should be provided if the syscall is missing too (in addition to the case where it's present but fails), so code doesn't end up with checks littered all over.

[therve]

Replying to jknight:

Not having looked at the patch yet, some test suggestions:

Thanks.

1) If the sendfile syscall is present, but fails, the code deals sanely (fallback to read/send, I suppose, or else some kind of notification?)

For now, that closes the connection.

1b) what if it fails because the file is unreadable or has an I/O error of some kind?

Same as above.

I'm interested to know what behavior we could choose.

2) SSL sockets don't support sendfile

Yes that's disabled, even thought I'm not really happy about the mecanism (value of the sendfile attribute)

3) it works, or at least, doesn't blow up on *BSD (which has a sendfile, with a different signature)

I didn't do that yet, but it's interesting.

[therve]

(In [22235]) Branching to 'sendfile-585-2'

[therve]

(In [22239]) Docs and cleanups.

Refs #585

[therve]

(In [23312]) Branching to 'sendfile-585-3'

[exarkun]

This is cool. I am going to call it a core ticket, though, since it's about extending the reactor interface.

[therve]

(In [28626]) Branching to 'sendfile-585-4'