[Twisted-Python] reactor for Linux io_uring

Adi Roiban adi at roiban.ro
Tue Jan 5 07:04:36 MST 2021


On Tue, 5 Jan 2021 at 13:44, Jean-Paul Calderone <exarkun at twistedmatrix.com>
wrote:

> On Tue, Jan 5, 2021 at 6:49 AM Barry Scott <barry.scott at forcepoint.com>
> wrote:
>
>> On Monday, 4 January 2021 04:01:42 GMT Ian Haywood wrote:
>> > In investigating async file I/O I came across this. In a nutshell it's
>> > the new epoll()
>>
>> > It's marginally more efficient although this is only apparent at very
>> > high loads.
>>
>> > What's more interesting is that io_uring accepts files as
>> > well as network/pipe handles: avoiding the need for threads.
>>
>> What threads? Why do you call out file FDs different from socket FDs?
>>
>> The point of io_uring is to avoid transitions between user and kernel
>> right?
>> Nothing to do with thread.
>>
>> In current twisted you can run complex network code without threads
>> already.
>>
>> > Here's a good intro: https://unixism.net/loti/index.html
>>
>> Also there is full coverage of io_uring on lwn.net.
>> Its a fast evolving kernel API.
>>
>> > If people think an IoUringReactor is worthwhile I'll open a ticket and
>> > make a start.
>>
>> I'm guessing that you will need to write a Python extension to get at
>> io_uring or use ctypes. Is that what you where expecting?
>>
>> > However it will need a reviewer... :-)
>>
>> Yes this is going to be complex code that few people have any experience
>> with.
>>
>
> Just so there's a counter-perspective out there, I would like to suggest
> that reactors are neither magical nor particular complex in the grand
> scheme of software and that if you start with the assumption that a piece
> of code is going to be complex and hard to review then you will almost
> certainly create a piece of software that is complex and hard to review.
>
> My recommendation would be to instead start with the assumption that it's
> just a bit more mundane code binding a relatively small and straightforward
> C API related to copying bytes from one location to another.  Yea, maybe
> there will be some hassles getting it to compile smoothly in all the
> desired environments or maybe there will be a few tricky areas for memory
> management or some other kind of low-level, localized bookkeeping - but
> those kinds of issues don't have to make for a complex piece of code.
> Starting from this assumption, try to produce an implementation that is
> straightforward, well-factored, and easily reviewed.  What do you have to
> lose?
>
> Jean-Paul
>
>
Hi,

There is this wrapper https://pypi.org/project/liburing/

I can give it a try and review a possible implementation.


My suggestion is to find a real world / production use case for the new
reactor so that we can run more than unit / functional tests.

-- 
Adi Roiban
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20210105/dc8df077/attachment.htm>


More information about the Twisted-Python mailing list