[Twisted-Python] Twisted, medusa, ZServer, and VFS's
abo at minkirri.apana.org.au
Wed Oct 10 01:48:59 EDT 2001
I've just been looking at Twisted Python for the past hour or two, read the
mailing list archives, and have some comments and questions.
I've been working on using Medusa for serving a virtual mirror via http and
ftp. I'm at the point where I'm close, but I'm starting to re-think some stuff,
in particular my choice of Medusa.
First my impressions of the three contenders; Twisted, Medusa, ZServer. Please
correct me if I'm wrong in the following summaries;
Medusa seems to be the daddy of them all. It's the oldest, which has benefits
and problems. It is a little messy from its evolution, but seems pretty mature.
It uses an async select loop to drive everything. It uses "asynchat" derived
class objects to communicate on sockets. Data can be sent by
pushing "producers" onto asynchat objects. Producers can be complex objects
that produce data, execute callbacks, whatever. It's ftp and http server
classes use a primative VFS to serve from. The asyncore and asynchat modules it
is built on are now part of Python.
Twisted seems to be a from-the-ground up re-invention of Medusa. It's newer,
but surprisingly it's bigger, dispite it's apparently less mature feature set.
It is similar in structure to Medusa, but simplifys it by dispensing with
producers. It can use a variety of event-loops, including Tk and GTK, or it's
own. It doesen't have a VFS (yet) so its ftp and http servers serve from the
underlying os filesystem.
ZServer grew out of Medusa. It uses the same basic underlying architecture, but
throws in threads to get around the problem of delayed producers blocking the
event loop. I'm not sure how tightly tied to Zope it is, but its http and ftp
servers generally serve from a ZODB database, presumably wrapped in a Medusa
VFS, though I have a feeling they might have changed that. ZServer also
supports webDAV serving. It is possible that some of the enhancements could be
merged back into Medusa, but probably it has changed so much it would be
My problem with Medusa is its http and ftp servers assume that the VFS can
deliver files wrapped in producers without blocking. I've fixed this by
creating a patch for Medusa's asynchat that adds support for a ready() method
to producers, so they can block without blocking the event loop. I'm currently
in the process of writing Medusa VFS's for ftp and eventualy http backends. In
the process I've also found that the Medusa ftp server is not as full featured
as I want.
In my search for Python VFS's I found PyVFS (http://www.pycage.de/). This is
modelled on the Gnome/MC VFS, so it
supports '/dir/somefile.tar.gz#tgz:/somepath' style paths to look inside tar,
tgz, ftp, whatever. The various different VFS backends are loaded dynamicly as
pluggins. These pluggins execute as a seperate process that are communicated
with over a channel. The API is too simplistic for me, with files
being "projected" out of the VFS to a local file to be manipulated/used. I
don't like the pluggin-process-channel architecture either.
In the reading of the Twisted mailing list, I saw a comment to the affect that
the Medusa VFS was an example of how _not_ to do it, which lead to using webDAV
as the API for a VFS. My gut feeling is DAV is a cool protocol for a VFS
backend, but I dunno about using it as the primary API. Sure, it supports meta-
data etc, but the reality is the API that is most widely used and understood is
the POSIX filesytem API, as exposed in Python by the os and os.path modules.
My solution for a VFS has been, upto now, based on Medusa's, but extending it
to be more like os and os.path. So far it's a filesystem class with most of the
os and os.path methods. One of the derived classes is a mountable_filesystem
that allows you to mount other VFS filesystems off it. At this point I'm
tempted to make a vfs module that emulates os and os.path so that you can mount
whatever vfs's you want first, and then just replace all your os.* calls with
vfs.* calls. Note that the one catch would be open() would need to be replaced
I'm sort of fishing for general suggestions, comments, and interest. I'm at the
point where I've just convinced myself my vfs is worth finishing, and my ready
() patch to asynchat is worth updating, but I'm not sure what to use as the
http and ftp server front-end, though I'm still leaning towards medusa. It
looks like Twister is not ready, and ZServer would be too hard to seperate from
PS... I'm not on the zope-dev list but I am on the twister and medusa lists.
The zope-dev list is too much non-ZServer stuff and that's all I'm interested
in. So zope-dev'ers, please reply to me or one/both of the other lists directly.
ABO: finger abo at minkirri.apana.org.au for more information.
More information about the Twisted-Python