[Twisted-Python] Async file I/O with Linux & Twisted

[Arnar Birgisson]

On Dec 20, 2007 6:16 PM, Michał Pasternak <michal.dtz at gmail.com> wrote:
> On Thu, 20 Dec 2007 09:23:43 +0000
> "Arnar Birgisson" <arnarbi at gmail.com> wrote:
>
> > Hi there,
> >
> > May I ask why you chose libaio over, for example, libevent?
>
> libevent is to C as Twisted is to Python. Both run on many
> OS/platforms, but neither give you asynchronous file read/write;
> read(2) on a file descriptor (not socket) will be always blocking.
> At least this is what I understood after quick glance at libevent
> source - all it does in buffer.c:evbuffer_read on non-win32 OS is
> calling read(2).

I'm pretty sure you are mistaken. As I understood, libevent provides a
portable API on top of whatever async I/O mechanisms there are on the
underlying system, selecting the best available implementation.

For Linux, I believe this it currently uses epoll, kqueue on freebsd,
or select() or poll() on older systems.

> libaio, instead, gives you async filesystem read/write access. Reading
> a few hundred megabytes of data won't block Twisted reactor, which is
> good.

libevent provides file and socket i/o, along with timers. Latest
version provides buffered i/o and async dns resolver and http server.

Please forgive (and ignore) me if I'm completely missing something :)

> Of course it would be cool to have Python libevent bindings - and even
> cooler it would be to have a portable libevent reactor in C - but would
> that really remove some of the limitations that Twisted has still to
> deal with? You'd still have to hack portable AIO implementation into
> libevent to have async file read/writes then ... :)

Python libevent bindings have existed for a while in pyevent, a pyrex
binding module:
http://code.google.com/p/pyevent/

Not sure how well it is maintained, but I've seen many references to
it so it looks like people are using it.

As for a libevent based reactor, there is an old ticket and branch for it here:
http://twistedmatrix.com/trac/ticket/1930
http://twistedmatrix.com/trac/browser/branches/libevent-1930-3

Googling for "twisted libevent reactor" gives many interesting results.

cheers,
Arnar