[Twisted-Python] FUSE

exarkun at twistedmatrix.com exarkun at twistedmatrix.com
Sun Apr 4 08:46:39 EDT 2010


On 03:21 am, donal.mcmullan at gmail.com wrote:
>I'm interested in doing some FUSE stuff with Twisted, mostly to
>prototype some ideas for profiling. Can anyone expand on Glyph's
>comment [1]:
>
>8< - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>The Python FUSE bindings obscure the issue because, unlike the C
>libfuse, they assume that your filesystem I/O is blocking, which
>severely limits the performance of python-based filesystems.  (You
>cannot receive more requests for I/O in your filesystem until the
>previous one has been completed with the pyfuse bindings, but you can 
>in
>C.)
>8< - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>
>I guess I'd like to know before I sink too much time into this, if
>there's an approach to Fuse/Twisted that would not have this
>limitation. This is a bit confused by the fact that there are three
>Fuse/Python bindings that I'm aware of: FusePython, FusePy, and S3QL
>[2].
>
>http://sourceforge.net/apps/mediawiki/fuse/index.php?title=FusePython
>http://code.google.com/p/fusepy/
>http://code.google.com/p/s3ql/
>
>If you've already been down this road, it'd be good to hear from you.

So, sitting at the bottom of libfuse is a file descriptor.  The kernel 
asks your process to service some filesystem operation by putting bytes 
into it that your process can read.  Your process services the requests 
by putting bytes into it that the kernel can read.

Each request/response pair is tagged, so there's no requirement that 
requests be responded to in the same order they're received.

A good Twisted-based solution would use the reactor to read and write 
that file descriptor (alongside anything else going on in the process) 
and support Deferreds (or at least asynchronous responses) to allow out- 
of-order responses and concurrent request handling.

I played around with fuse a little bit to implement something like sshfs 
a while ago.  I did _not_ use the approach described above, because it 
was easier to use something from Armin Rigo's sandbox in a thread, 
dispatching request handling into Twisted, to achieve the necessary 
concurrency and so forth.  You can find that code here:

http://twistedmatrix.com/trac/browser/sandbox/exarkun/sshfs.py?rev=26941

It's just something I was messing around with.  I'm sure you'll find 
lots of problems with it (I know of at least one).  It might help with 
part of the general idea, though.  To get rid of the use of threads 
(which is a good idea for a real tool), you just need to port Armin's 
parsing code into a Twisted Protocol class and hook it up to a 
FileDescriptor of some sort that reads and writes the fuse fd.

Jean-Paul



More information about the Twisted-Python mailing list