[Twisted-Python] Async file I/O with Linux & Twisted

Arnar Birgisson arnarbi at gmail.com
Thu Dec 20 04:23:43 EST 2007


Hi there,

May I ask why you chose libaio over, for example, libevent?

Arnar

On Dec 20, 2007 2:42 AM, Michał Pasternak <michal.dtz at gmail.com> wrote:
> Hi,
>
> I've spent last few days trying to write a wrapper for libaio.
>
> Libaio is a simple Linux-only library, which should in theory support
> async read/write on a file descriptor. In practice - this depends on
> many strange requirements and the documentation is basically
> nonexistent (if you don't count lwn articles and kernel mailing list
> archives), so that last few days of C coding looked like this: (it's
> funny how every C coding session looks similar)
>
>         http://youtube.com/watch?v=-JbhMvzeX7s
>
> Just to summarize...
>  * no AIO reads from socket
>  * not every filesystem supported
>  * output buffer must be N-pages long and aligned to page start
>  * file access is unbuffered (yep, no cache!)
>  * ... and it may still - *silently* - block in some circumstances
>    (like, when you run out of "block layer requests", like if
>    I knew, what are those)
>
> Anyways, libaio seems to be a cool idea - maybe someday kernel guys
> will do some more work on it (like, commit those patches which add
> buffer aio) and it will be possible to do AIO stuff on sockets. On the
> other hand, I found no way other to be informed about data
> availability, than periodically reaping events (want libaio and epoll
> integration? use some more kernel patches...)
>
> You can get the module code and read about current linux aio problems
> here:
>
>         http://code.google.com/p/twisted-linux-aio/
>
> It is not a general-purpose module - it's tightly tied to Twisted.
> Only reads are supported ATM and there's no Python queue, only that low
> level one (see TODO.txt). If you're lucky, you may even see the proof
> of concept code running - w/o blocking. There are more things to come,
> especially that async shutil.copyfileobj replacement... :)
>
> Sad thing is, that Linux seems to lack much in this area, especially
> when compared to FreeBSD...
>
> --
> m
>
> PS: I feel glad I didn't started with POSIX aio_* implementation - on
> Linux it is said to launch a new thread for every fd request!
> http://www.atnf.csiro.au/people/rgooch/linux/docs/io-events.html - way
> to go, glibc programmers! I hope things are looking better now, than
> they were that few years ago...
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>


More information about the Twisted-Python mailing list