[Twisted-Python] reactor for Linux io_uring

[Barry Scott]

On Tuesday, 5 January 2021 13:31:49 GMT Jean-Paul Calderone wrote:
> On Tue, Jan 5, 2021 at 6:49 AM Barry Scott <barry.scott at forcepoint.com>
> wrote:
> 
> > On Monday, 4 January 2021 04:01:42 GMT Ian Haywood wrote:
> > > In investigating async file I/O I came across this. In a nutshell it's
> > > the new epoll()
> >
> > > It's marginally more efficient although this is only apparent at very
> > > high loads.
> >
> > > What's more interesting is that io_uring accepts files as
> > > well as network/pipe handles: avoiding the need for threads.
> >
> > What threads? Why do you call out file FDs different from socket FDs?
> >
> > The point of io_uring is to avoid transitions between user and kernel
> > right?
> > Nothing to do with thread.
> >
> > In current twisted you can run complex network code without threads
> > already.
> >
> > > Here's a good intro: https://unixism.net/loti/index.html
> >
> > Also there is full coverage of io_uring on lwn.net.
> > Its a fast evolving kernel API.
> >
> > > If people think an IoUringReactor is worthwhile I'll open a ticket and
> > > make a start.
> >
> > I'm guessing that you will need to write a Python extension to get at
> > io_uring or use ctypes. Is that what you where expecting?
> >
> > > However it will need a reviewer... :-)
> >
> > Yes this is going to be complex code that few people have any experience
> > with.
> >
> 
> Just so there's a counter-perspective out there, I would like to suggest
> that reactors are neither magical nor particular complex in the grand
> scheme of software and that if you start with the assumption that a piece
> of code is going to be complex and hard to review then you will almost
> certainly create a piece of software that is complex and hard to review.
>
> My recommendation would be to instead start with the assumption that it's
> just a bit more mundane code binding a relatively small and straightforward
> C API related to copying bytes from one location to another.  Yea, maybe
> there will be some hassles getting it to compile smoothly in all the
> desired environments or maybe there will be a few tricky areas for memory
> management or some other kind of low-level, localized bookkeeping - but
> those kinds of issues don't have to make for a complex piece of code.
> Starting from this assumption, try to produce an implementation that is
> straightforward, well-factored, and easily reviewed.  What do you have to
> lose?

I'm not against doing this and would love to see a PoC.
I've been following the io_uring work with great interest.

But my experience with things like this inside kernels leads me to expect
complexity.

Barry

> 
> Jean-Paul
> 
> 
> >
> > Barry
> >
> > > Ian
> > >
> > > _______________________________________________
> > > Twisted-Python mailing list
> > > Twisted-Python at twistedmatrix.com
> > > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> > >
> >
> >
> >
> >
> > _______________________________________________
> > Twisted-Python mailing list
> > Twisted-Python at twistedmatrix.com
> > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> >
>