[Twisted-Python] reactor for Linux io_uring

[Tobias Oberstein]

Am 13.03.21 um 01:01 schrieb Ian Haywood:
> On 12/03/2021 8:29 pm, Tobias Oberstein wrote:
>> since with io_uring the queue/ring to append to never blocks for 
>> writes, that buffering in twisted would lead to double buffering (as 
>> the ring already buffers)
>>
>> then, io_uring follows a completion IO model (as IOCP). quite 
>> different from the "ready to write/read" model (as in epoll etc).
>>
>> I am wondering, do you already have a design for these aspects of 
>> io_uring / twisted?
> I haven't thought about it in detail, but python presents mmap as 
> bytearray, so I would have thought it fairly easy to write directly to 
> the ring from Protocol.write

fwiw, couple of thoughts:

to make it "zero copy" right up into python, a couple of things would be 
needed (there might be other ways of course ..):

the memory to write into on the producer side must be allocated and 
managed via liburing

that memory might for example actually reside inside a NIC and be DMA'ed 
into main memory ("zero CPU copy") or even directly mapped into the user 
level address space ("true zero copy")

even when taking the very basic case of just transmitting an unstructed 
bunch of bytes, to make it truly zero copy up into python, sth like

https://zero-buffer.readthedocs.io

might be needed rather than standard python str/bytes.

then, for structured data, if zero-buffer would be used in python lib of 
flatbuffers for str/bytes

https://google.github.io/flatbuffers/

this would result in a truly zero-copy transfer _and_ serialization for 
native structured objects.

all of that with zero/minimal syscalls.

in a way, the holy grail;)

cheers,
/Tobias

alternatives for flatbuffers:
https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html