[Twisted-Python] Twisted, medusa, ZServer, and VFS's

Glyph Lefkowitz glyph at twistedmatrix.com
Wed Oct 10 04:56:38 EDT 2001


I'm only responding to the twisted-python list, since the cross-post 
seemed excessive.  Feel free to rebroadcast this if ensuing discussion 
on other lists is interesting.

On Wednesday, October 10, 2001, at 12:48  AM, Donovan Baarda wrote:

> G'day,
>
> I've just been looking at Twisted Python for the past hour or two, read 
> the
> mailing list archives, and have some comments and questions.

Thank you for your interest.  Here's a twisted way of looking at your 
questions :-)

> First my impressions of the three contenders; Twisted, Medusa, ZServer. 
> Please
> correct me if I'm wrong in the following summaries;
>
> Medusa seems to be the daddy of them all.

I don't see what you mean by that; about the only things that Medusa and 
Twisted share is that they are both asynchronous networking frameworks.  
They seem to have a fairly different approach to how protocols are 
written and integrated, and in what their scope for future development 
is.

> It's the oldest, which has benefits and problems. It is a little messy 
> from its evolution, but seems pretty mature. It uses an async select 
> loop to drive everything. It uses "asynchat" derived class objects to 
> communicate on sockets. Data can be sent by pushing "producers" onto 
> asynchat objects. Producers can be complex objects that produce data, 
> execute callbacks, whatever. It's ftp and http server classes use a 
> primative VFS to serve from. The asyncore and asynchat modules it is 
> built on are now part of Python.

This sounds pretty accurate to me.

> Twisted seems to be a from-the-ground up re-invention of Medusa.

As Moshe said, only insofar as it's a from-the-ground-up re-invention of 
about 6 or 7 other things.

> It's newer, but surprisingly it's bigger, dispite it's apparently less 
> mature feature set.

What do you mean when you say "less mature"?  Twisted's features have 
been around for less time (hence, "newer") but compare, for example, 
twisted.spread.pb with rpc_server.  Or twisted.words with chat_server  
Would you characterize the medusa approach in any of these comparisons 
as "more mature"?

What is the Medusa equivalent of twisted.reality, twisted.mail, 
twisted.web.widgets, or twisted.enterprise?  These services are at 
varying levels of maturity, but surely the fact that they exist at all 
has to count for something :-).

> It is similar in structure to Medusa,

At some extremely superficial level, I guess this is true.  However, 
Twisted does a lot more than just clone medusa.  Even at a basic level, 
you could say that it complicates the medusa structure a great deal with 
a unified notion of authentication, automatic persistence, and 
incidentally, several full-featured applications. :-)

> but simplifys it by dispensing with producers.

Twisted has producers, but only when you need them.

http://twistedmatrix.com/users/glyph/TwistedDocs/Twisted-0.11.0/twisted/internet/
abstract_FileDescriptor.py.html#registerProducer

> It can use a variety of event-loops, including Tk and GTK, or it's own. 
> It doesen't have a VFS (yet) so its ftp and http servers serve from the 
> underlying os filesystem.

Well, it depends what you mean by VFS.  Twisted has a perception of the 
filesystem as more like a special-case of "container" than containers as 
a special-case of the filesystem.  There are containers which can 
respond to specifics of the HTTP protocol that are not derived from 
files; would you call that part of a "VFS"?  The semantics of 
"__builtins__.open" are not sufficiently rich to support that.

> In my search for Python VFS's I found PyVFS (http://www.pycage.de/). 
> This is modelled on the Gnome/MC VFS, so it supports 
> '/dir/somefile.tar.gz#tgz:/somepath' style paths to look inside tar, 
> tgz, ftp, whatever. The various different VFS backends are loaded 
> dynamicly as pluggins. These pluggins execute as a seperate process 
> that are communicated with over a channel. The API is too simplistic 
> for me, with files being "projected" out of the VFS to a local file to 
> be manipulated/used. I don't like the pluggin-process-channel 
> architecture either.

Why not?  It sounds clever to me.  I don't have a good picture of your 
requirements (other than "HTTP and FTP" at this point, so I can't fathom 
why you like or don't like this particular solution.  (Why is it 
relevant?)

> In the reading of the Twisted mailing list, I saw a comment to the 
> affect that
> the Medusa VFS was an example of how _not_ to do it, which lead to 
> using webDAV
> as the API for a VFS. My gut feeling is DAV is a cool protocol for a VFS
> backend, but I dunno about using it as the primary API. Sure, it 
> supports meta-
> data etc, but the reality is the API that is most widely used and 
> understood is
> the POSIX filesytem API, as exposed in Python by the os and os.path 
> modules.

Yes, but that's a blocking API; ergo, it does not work in an 
asynchronous framework like Twisted.  Not all "files" are associated 
with a file descriptor, so they may potentially support different 
operations.  Directory listing on FTP, HTTP, and WebDAV sites is not 
necessarily consistent with the files that are actually available.  
There is metadata associated with some requests, and not with others... 
in short, there are lots of subtle issues involved with supporting each 
of these types of hierarchies well, and a blanket virtual "filesystem" 
implementation does not satisfy all (or even a reasonably large subset) 
of them.  For different protocols, there may be API differences, unless 
some of the protocols are stripped to the "lowest common denominator", 
e.g. POSIX.

> My solution for a VFS has been, upto now, based on Medusa's, but 
> extending it
> to be more like os and os.path. So far it's a filesystem class with 
> most of the
> os and os.path methods. One of the derived classes is a 
> mountable_filesystem
> that allows you to mount other VFS filesystems off it. At this point I'm
> tempted to make a vfs module that emulates os and os.path so that you 
> can mount
> whatever vfs's you want first, and then just replace all your os.* 
> calls with
> vfs.* calls. Note that the one catch would be open() would need to be 
> replaced
> with vfs.open().

Not necessarily.  You could always hack up the __builtins__ module at 
runtime to point to your newer, better open().  Either way it seems like 
there are probably issues with security & the presence of *real* file 
descriptors that you have to think about...

> I'm sort of fishing for general suggestions, comments, and interest. 
> I'm at the
> point where I've just convinced myself my vfs is worth finishing, and 
> my ready
> () patch to asynchat is worth updating, but I'm not sure what to use as 
> the
> http and ftp server front-end, though I'm still leaning towards medusa. 
> It
> looks like Twister is not ready, and ZServer would be too hard to 
> seperate from
> Zope.

Ready for what, is the question? :-)

Short form: HTTP yes, FTP no (but it could be with a little work), 
everything else yes.

Long form:

We do not currently have (or have any high-priority plans to produce) a 
"VFS", but I would contend that such a system is not necessary when you 
look at the way that Twisted does web resources.  FTP doesn't currently 
use that model, but I don't think it would be a difficult modification; 
it uses the same Producer model that HTTP does.  I estimate that it 
would be easier to modify Twisted in this way than to undertake a 
project to do your own VFS, but since I'm not exactly sure what's going 
to make you happy, I don't have a high degree of confidence in that 
estimation.

As far as robustness (which is an implied issue with Twisted's 
"maturity"), Twisted is being used to run the main twistedmatrix.com 
site.  We always run the most recent development version.  So far, it's 
only served 17.422 downloads of Twisted itself and approximately three 
quarters of a million hits, and survived five or six freshmeat 
"attacks" :-).  It did crash, once, about 8 months ago after a major 
refactoring (before we had acceptance tests...), but that bug was 
quickly fixed.  We don't run a public FTP server, but the core parts of 
Twisted that enable one to run a website where you can download files 
are very personally important to me, so draw your own conclusions :-).

--
______      you are in a maze of twisted little applications, all
|   |_\     remarkably consistent.
|     |          -- glyph lefkowitz, glyph @ twisted matrix . com
|_____|             http://www.twistedmatrix.com/





More information about the Twisted-Python mailing list