#4387 enhancement new
API for distributing `ITransport` and `IListeningPort` implementations to different processes
|Reported by:||Glyph||Owned by:|
|Cc:||thomasvs, Marc-André Lureau, bra, Manish||Branch:|
One design pattern for high-performance servers is to have a master process which accepts incoming connections, but then dispatches those connections to sub-processes or peer processes which handle the actual content of the connection, either immediately or after some initial handshake.
While it's always possible to simply forward traffic, it is of course more efficient for the master accepting process to just get out of the way completely and let the application process handle all system calls and buffering itself.
On Windows, I believe the WSADuplicateSocket function serves the same purpose, albeit in a more restricted form (only sockets may be transferred, not arbitrary handles).
You can already implement bits and pieces of this using Twisted, by inspecting internal state in the reactor, but we should have a high-level, portable interface for establishing the channel to send sockets to a different process.
While I tend to prefer my own implementation for its minimalist usage of C (fluendo implements FD-sending at the C level, whereas I just have a very literal-minded
sendmsg wrapper, with FD-sending implemented in Python) it isn't quite general enough to use as-is. At least, the high-level bits aren't, as they rely on application-level (protocol-level) hooks to report status back to the parent process, whereas that should be the responsibility of the transport. However, as I said, the lowest-level bits are a fairly straightforward binding to the
recvmsg C library functions; essentially a marginally improved rewrite of the sendmsg code which has been floating around in various people's sandboxes for quite a while.
While I mention the Windows here to provide a reference point for the portability restrictions of a high-level API, we can definitely defer that work to a separate ticket.
Also, given that you can inherit the connection in multiple different ways - via file descriptor inheritance, via
connect() to a numeric UNIX socket,
connect() to a UNIX named socket, or via a process identifier on Windows, we may need to provide some platform-specific bootstrapping APIs which result in a common interface once invoked.