[Twisted-Python] Twisted, medusa, ZServer, and VFS's

Donovan Baarda abo at minkirri.apana.org.au
Wed Oct 10 09:38:48 EDT 2001

On Wed, Oct 10, 2001 at 03:56:38AM -0500, Glyph Lefkowitz wrote:
> I'm only responding to the twisted-python list, since the cross-post 
> seemed excessive.  Feel free to rebroadcast this if ensuing discussion 
> on other lists is interesting.

sticking to this convention... :-)

> On Wednesday, October 10, 2001, at 12:48  AM, Donovan Baarda wrote:
> They seem to have a fairly different approach to how protocols are 
> written and integrated, and in what their scope for future development 
> is.

This I guess is part of what is putting me off Twisted. I'm familiar with
Medusa, and Twisted is just different enough that I've got a learning curve
ahead of me. Unless I get convinced Twisted is worth the extra effort...

> > Twisted seems to be a from-the-ground up re-invention of Medusa.
> As Moshe said, only insofar as it's a from-the-ground-up re-invention of 
> about 6 or 7 other things.

Python async socket framework... it's filling the same void :-)

> > It's newer, but surprisingly it's bigger, dispite it's apparently less 
> > mature feature set.
> What do you mean when you say "less mature"?  Twisted's features have 
> been around for less time (hence, "newer") but compare, for example, 
> twisted.spread.pb with rpc_server.  Or twisted.words with chat_server  
> Would you characterize the medusa approach in any of these comparisons 
> as "more mature"?
> What is the Medusa equivalent of twisted.reality, twisted.mail, 
> twisted.web.widgets, or twisted.enterprise?  These services are at 
> varying levels of maturity, but surely the fact that they exist at all 
> has to count for something :-).

That probably accounts for Twisted having more code. However, in my case I'm
just after http and ftp server capabilities. I have a feeling Medusa's ftp
code at least is more complete.

> > It is similar in structure to Medusa,
> At some extremely superficial level, I guess this is true.  However, 
> Twisted does a lot more than just clone medusa.  Even at a basic level, 
> you could say that it complicates the medusa structure a great deal with 
> a unified notion of authentication, automatic persistence, and 
> incidentally, several full-featured applications. :-)

I'm also a bit of a less-is-more person... I don't really need all of that.
However, if the framework is neat, and I get that without extra hassles and
bloat, I guess I'd use it.

> > but simplifys it by dispensing with producers.
> Twisted has producers, but only when you need them.
> http://twistedmatrix.com/users/glyph/TwistedDocs/Twisted-0.11.0/twisted/internet/
> abstract_FileDescriptor.py.html#registerProducer

Hmmm... looking into this.

> > It can use a variety of event-loops, including Tk and GTK, or it's own. 
> > It doesen't have a VFS (yet) so its ftp and http servers serve from the 
> > underlying os filesystem.
> Well, it depends what you mean by VFS.  Twisted has a perception of the 
> filesystem as more like a special-case of "container" than containers as 
> a special-case of the filesystem.  There are containers which can 
> respond to specifics of the HTTP protocol that are not derived from 
> files; would you call that part of a "VFS"?  The semantics of 
> "__builtins__.open" are not sufficiently rich to support that.

I was just looking at the ftp part, and it seems that it can only serve
files. If it can to more than this, I guess I'm more interested.

> > In my search for Python VFS's I found PyVFS (http://www.pycage.de/). 
> > This is modelled on the Gnome/MC VFS, so it supports 
> > '/dir/somefile.tar.gz#tgz:/somepath' style paths to look inside tar, 
> > tgz, ftp, whatever. The various different VFS backends are loaded 
> > dynamicly as pluggins. These pluggins execute as a seperate process 
> > that are communicated with over a channel. The API is too simplistic 
> > for me, with files being "projected" out of the VFS to a local file to 
> > be manipulated/used. I don't like the pluggin-process-channel 
> > architecture either.
> Why not?  It sounds clever to me.  I don't have a good picture of your 
> requirements (other than "HTTP and FTP" at this point, so I can't fathom 
> why you like or don't like this particular solution.  (Why is it 
> relevant?)

It is overkill for what it is. Spawning whole extra processes and using
inter-process communication over a channel when just classes and/or
threads would do the job.

The biggest limitation was the API... I basicly want to serve up a virtual
mirror, which means I need to be able to identify and mirror things like

However, because it is quite simple, I was thinking of making a VFS backend
for my VFS that talks to PyVFS... more as a proof of concept, but also to
get tar.gz, cpio, mailbox, etc for free.

> > In the reading of the Twisted mailing list, I saw a comment to the 
> > affect that
> > the Medusa VFS was an example of how _not_ to do it, which lead to 
> > using webDAV
> > as the API for a VFS. My gut feeling is DAV is a cool protocol for a VFS
> > backend, but I dunno about using it as the primary API. Sure, it 
> > supports meta-
> > data etc, but the reality is the API that is most widely used and 
> > understood is
> > the POSIX filesytem API, as exposed in Python by the os and os.path 
> > modules.
> Yes, but that's a blocking API; ergo, it does not work in an 
> asynchronous framework like Twisted.  Not all "files" are associated 

The Medusa VFS is a stripped down POSIX, and it actualy causes blocking
problems for my application. That's why I added ready() support to asynchat,
so that file producers can tell the select loop they would block and be
excluded from that time round the loop. This is basicly a hack... a probably
neater way is to make the file producers asychat's themselves but that would
require major restructuring of Medusa's ftp and http servers.

> with a file descriptor, so they may potentially support different 
> operations.  Directory listing on FTP, HTTP, and WebDAV sites is not 
> necessarily consistent with the files that are actually available.  
> There is metadata associated with some requests, and not with others... 
> in short, there are lots of subtle issues involved with supporting each 
> of these types of hierarchies well, and a blanket virtual "filesystem" 
> implementation does not satisfy all (or even a reasonably large subset) 
> of them.  For different protocols, there may be API differences, unless 
> some of the protocols are stripped to the "lowest common denominator", 
> e.g. POSIX.

That's why I'm going for POSIX :-)

> > My solution for a VFS has been, upto now, based on Medusa's, but 
> > extending it
> > to be more like os and os.path. So far it's a filesystem class with 
> > most of the
> > os and os.path methods. One of the derived classes is a 
> > mountable_filesystem
> > that allows you to mount other VFS filesystems off it. At this point I'm
> > tempted to make a vfs module that emulates os and os.path so that you 
> > can mount
> > whatever vfs's you want first, and then just replace all your os.* 
> > calls with
> > vfs.* calls. Note that the one catch would be open() would need to be 
> > replaced
> > with vfs.open().
> Not necessarily.  You could always hack up the __builtins__ module at 
> runtime to point to your newer, better open().  Either way it seems like 
> there are probably issues with security & the presence of *real* file 
> descriptors that you have to think about...

Thats why I'd keep it seperate inside a vfs module... remember at least one
vfs backend would be using the builtin open and os modules to access the
real fs. Though now you mention it... it could be away of transperantly
running any application on top of a VFS without changing it at all. Hmmm...
bound to be clashes... I wonder...

> > I'm sort of fishing for general suggestions, comments, and interest. 
> > I'm at the
> > point where I've just convinced myself my vfs is worth finishing, and 
> > my ready
> > () patch to asynchat is worth updating, but I'm not sure what to use as 
> > the
> > http and ftp server front-end, though I'm still leaning towards medusa. 
> > It
> > looks like Twister is not ready, and ZServer would be too hard to 
> > seperate from
> > Zope.
> Ready for what, is the question? :-)
> Short form: HTTP yes, FTP no (but it could be with a little work), 
> everything else yes.

FTP is the main one...though I'd really love rsync... roll on librsync :-)

> Long form:
> We do not currently have (or have any high-priority plans to produce) a 
> "VFS", but I would contend that such a system is not necessary when you 
> look at the way that Twisted does web resources.  FTP doesn't currently 
> use that model, but I don't think it would be a difficult modification; 
> it uses the same Producer model that HTTP does.  I estimate that it 
> would be easier to modify Twisted in this way than to undertake a 
> project to do your own VFS, but since I'm not exactly sure what's going 
> to make you happy, I don't have a high degree of confidence in that 
> estimation.

The application is basicly a mirror daemon that serves up a virtual mirror
of an ftp, rsync or http site. To the ftp and http clients, it appears to be
a full mirror. Files are fetched on demand and stored in a partial mirror on
the server.

My plan was to use a VFS to 'mount' the remote ftp, rsync, or http site. The
partial mirror on the server would also be accessed through the same VFS
interface. Then I was going to overlay a "mirrorfs" VFS over them both, that
would mirror the remote VFS to the local one on demand. The http/ftp server
part would then just serve files from the mirrorfs VFS.

The beauty of this is the different VFS backends would allow you to do wierd
things like on-demand mirror a remote ftp server into a local tar.gz file,
not that you'd want to :-)

ABO: finger abo at minkirri.apana.org.au for more info, including pgp key

More information about the Twisted-Python mailing list