[Twisted-Python] Twisted, medusa, ZServer, and VFS's

[Donovan Baarda]

G'day,

I've just been looking at Twisted Python for the past hour or two, read the 
mailing list archives, and have some comments and questions.

I've been working on using Medusa for serving a virtual mirror via http and 
ftp. I'm at the point where I'm close, but I'm starting to re-think some stuff, 
in particular my choice of Medusa.

First my impressions of the three contenders; Twisted, Medusa, ZServer. Please 
correct me if I'm wrong in the following summaries;

Medusa seems to be the daddy of them all. It's the oldest, which has benefits 
and problems. It is a little messy from its evolution, but seems pretty mature. 
It uses an async select loop to drive everything. It uses "asynchat" derived 
class objects to communicate on sockets. Data can be sent by 
pushing "producers" onto asynchat objects. Producers can be complex objects 
that produce data, execute callbacks, whatever. It's ftp and http server 
classes use a primative VFS to serve from. The asyncore and asynchat modules it 
is built on are now part of Python.

Twisted seems to be a from-the-ground up re-invention of Medusa. It's newer, 
but surprisingly it's bigger, dispite it's apparently less mature feature set. 
It is similar in structure to Medusa, but simplifys it by dispensing with 
producers. It can use a variety of event-loops, including Tk and GTK, or it's 
own. It doesen't have a VFS (yet) so its ftp and http servers serve from the 
underlying os filesystem.

ZServer grew out of Medusa. It uses the same basic underlying architecture, but 
throws in threads to get around the problem of delayed producers blocking the 
event loop. I'm not sure how tightly tied to Zope it is, but its http and ftp 
servers generally serve from a ZODB database, presumably wrapped in a Medusa 
VFS, though I have a feeling they might have changed that. ZServer also 
supports webDAV serving. It is possible that some of the enhancements could be 
merged back into Medusa, but probably it has changed so much it would be 
difficult.

My problem with Medusa is its http and ftp servers assume that the VFS can 
deliver files wrapped in producers without blocking. I've fixed this by 
creating a patch for Medusa's asynchat that adds support for a ready() method 
to producers, so they can block without blocking the event loop. I'm currently 
in the process of writing Medusa VFS's for ftp and eventualy http backends. In 
the process I've also found that the Medusa ftp server is not as full featured 
as I want.

In my search for Python VFS's I found PyVFS (http://www.pycage.de/). This is 
modelled on the Gnome/MC VFS, so it 
supports '/dir/somefile.tar.gz#tgz:/somepath' style paths to look inside tar, 
tgz, ftp, whatever. The various different VFS backends are loaded dynamicly as 
pluggins. These pluggins execute as a seperate process that are communicated 
with over a channel. The API is too simplistic for me, with files 
being "projected" out of the VFS to a local file to be manipulated/used. I 
don't like the pluggin-process-channel architecture either.

In the reading of the Twisted mailing list, I saw a comment to the affect that 
the Medusa VFS was an example of how _not_ to do it, which lead to using webDAV 
as the API for a VFS. My gut feeling is DAV is a cool protocol for a VFS 
backend, but I dunno about using it as the primary API. Sure, it supports meta-
data etc, but the reality is the API that is most widely used and understood is 
the POSIX filesytem API, as exposed in Python by the os and os.path modules.

My solution for a VFS has been, upto now, based on Medusa's, but extending it 
to be more like os and os.path. So far it's a filesystem class with most of the 
os and os.path methods. One of the derived classes is a mountable_filesystem 
that allows you to mount other VFS filesystems off it. At this point I'm 
tempted to make a vfs module that emulates os and os.path so that you can mount 
whatever vfs's you want first, and then just replace all your os.* calls with 
vfs.* calls. Note that the one catch would be open() would need to be replaced 
with vfs.open().

I'm sort of fishing for general suggestions, comments, and interest. I'm at the 
point where I've just convinced myself my vfs is worth finishing, and my ready
() patch to asynchat is worth updating, but I'm not sure what to use as the 
http and ftp server front-end, though I'm still leaning towards medusa. It 
looks like Twister is not ready, and ZServer would be too hard to seperate from 
Zope.

PS... I'm not on the zope-dev list but I am on the twister and medusa lists. 
The zope-dev list is too much non-ZServer stuff and that's all I'm interested 
in. So zope-dev'ers, please reply to me or one/both of the other lists directly.

--
ABO: finger abo at minkirri.apana.org.au for more information.