[Twisted-Python] Async file I/O with Linux & Twisted

[Michał Pasternak]

Hi, 

I've spent last few days trying to write a wrapper for libaio.

Libaio is a simple Linux-only library, which should in theory support
async read/write on a file descriptor. In practice - this depends on
many strange requirements and the documentation is basically
nonexistent (if you don't count lwn articles and kernel mailing list
archives), so that last few days of C coding looked like this: (it's
funny how every C coding session looks similar)

	http://youtube.com/watch?v=-JbhMvzeX7s
	
Just to summarize...
 * no AIO reads from socket 
 * not every filesystem supported 
 * output buffer must be N-pages long and aligned to page start
 * file access is unbuffered (yep, no cache!)
 * ... and it may still - *silently* - block in some circumstances
   (like, when you run out of "block layer requests", like if 
   I knew, what are those)

Anyways, libaio seems to be a cool idea - maybe someday kernel guys
will do some more work on it (like, commit those patches which add
buffer aio) and it will be possible to do AIO stuff on sockets. On the
other hand, I found no way other to be informed about data
availability, than periodically reaping events (want libaio and epoll
integration? use some more kernel patches...)

You can get the module code and read about current linux aio problems
here:

	http://code.google.com/p/twisted-linux-aio/

It is not a general-purpose module - it's tightly tied to Twisted.
Only reads are supported ATM and there's no Python queue, only that low
level one (see TODO.txt). If you're lucky, you may even see the proof
of concept code running - w/o blocking. There are more things to come,
especially that async shutil.copyfileobj replacement... :)

Sad thing is, that Linux seems to lack much in this area, especially
when compared to FreeBSD...

-- 
m

PS: I feel glad I didn't started with POSIX aio_* implementation - on
Linux it is said to launch a new thread for every fd request!
http://www.atnf.csiro.au/people/rgooch/linux/docs/io-events.html - way
to go, glibc programmers! I hope things are looking better now, than
they were that few years ago...