[Twisted-Python] reactor for Linux io_uring

[Jean-Paul Calderone]

On Tue, Jan 5, 2021 at 6:49 AM Barry Scott <barry.scott at forcepoint.com>
wrote:

> On Monday, 4 January 2021 04:01:42 GMT Ian Haywood wrote:
> > In investigating async file I/O I came across this. In a nutshell it's
> > the new epoll()
>
> > It's marginally more efficient although this is only apparent at very
> > high loads.
>
> > What's more interesting is that io_uring accepts files as
> > well as network/pipe handles: avoiding the need for threads.
>
> What threads? Why do you call out file FDs different from socket FDs?
>
> The point of io_uring is to avoid transitions between user and kernel
> right?
> Nothing to do with thread.
>
> In current twisted you can run complex network code without threads
> already.
>
> > Here's a good intro: https://unixism.net/loti/index.html
>
> Also there is full coverage of io_uring on lwn.net.
> Its a fast evolving kernel API.
>
> > If people think an IoUringReactor is worthwhile I'll open a ticket and
> > make a start.
>
> I'm guessing that you will need to write a Python extension to get at
> io_uring or use ctypes. Is that what you where expecting?
>
> > However it will need a reviewer... :-)
>
> Yes this is going to be complex code that few people have any experience
> with.
>

Just so there's a counter-perspective out there, I would like to suggest
that reactors are neither magical nor particular complex in the grand
scheme of software and that if you start with the assumption that a piece
of code is going to be complex and hard to review then you will almost
certainly create a piece of software that is complex and hard to review.

My recommendation would be to instead start with the assumption that it's
just a bit more mundane code binding a relatively small and straightforward
C API related to copying bytes from one location to another.  Yea, maybe
there will be some hassles getting it to compile smoothly in all the
desired environments or maybe there will be a few tricky areas for memory
management or some other kind of low-level, localized bookkeeping - but
those kinds of issues don't have to make for a complex piece of code.
Starting from this assumption, try to produce an implementation that is
straightforward, well-factored, and easily reviewed.  What do you have to
lose?

Jean-Paul


>
> Barry
>
> > Ian
> >
> > _______________________________________________
> > Twisted-Python mailing list
> > Twisted-Python at twistedmatrix.com
> > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> >
>
>
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20210105/ac1f399c/attachment.htm>