[Twisted-Python] Twisted, medusa, ZServer, and VFS's
glyph at twistedmatrix.com
Wed Oct 10 04:56:38 EDT 2001
I'm only responding to the twisted-python list, since the cross-post
seemed excessive. Feel free to rebroadcast this if ensuing discussion
on other lists is interesting.
On Wednesday, October 10, 2001, at 12:48 AM, Donovan Baarda wrote:
> I've just been looking at Twisted Python for the past hour or two, read
> mailing list archives, and have some comments and questions.
Thank you for your interest. Here's a twisted way of looking at your
> First my impressions of the three contenders; Twisted, Medusa, ZServer.
> correct me if I'm wrong in the following summaries;
> Medusa seems to be the daddy of them all.
I don't see what you mean by that; about the only things that Medusa and
Twisted share is that they are both asynchronous networking frameworks.
They seem to have a fairly different approach to how protocols are
written and integrated, and in what their scope for future development
> It's the oldest, which has benefits and problems. It is a little messy
> from its evolution, but seems pretty mature. It uses an async select
> loop to drive everything. It uses "asynchat" derived class objects to
> communicate on sockets. Data can be sent by pushing "producers" onto
> asynchat objects. Producers can be complex objects that produce data,
> execute callbacks, whatever. It's ftp and http server classes use a
> primative VFS to serve from. The asyncore and asynchat modules it is
> built on are now part of Python.
This sounds pretty accurate to me.
> Twisted seems to be a from-the-ground up re-invention of Medusa.
As Moshe said, only insofar as it's a from-the-ground-up re-invention of
about 6 or 7 other things.
> It's newer, but surprisingly it's bigger, dispite it's apparently less
> mature feature set.
What do you mean when you say "less mature"? Twisted's features have
been around for less time (hence, "newer") but compare, for example,
twisted.spread.pb with rpc_server. Or twisted.words with chat_server
Would you characterize the medusa approach in any of these comparisons
as "more mature"?
What is the Medusa equivalent of twisted.reality, twisted.mail,
twisted.web.widgets, or twisted.enterprise? These services are at
varying levels of maturity, but surely the fact that they exist at all
has to count for something :-).
> It is similar in structure to Medusa,
At some extremely superficial level, I guess this is true. However,
Twisted does a lot more than just clone medusa. Even at a basic level,
you could say that it complicates the medusa structure a great deal with
a unified notion of authentication, automatic persistence, and
incidentally, several full-featured applications. :-)
> but simplifys it by dispensing with producers.
Twisted has producers, but only when you need them.
> It can use a variety of event-loops, including Tk and GTK, or it's own.
> It doesen't have a VFS (yet) so its ftp and http servers serve from the
> underlying os filesystem.
Well, it depends what you mean by VFS. Twisted has a perception of the
filesystem as more like a special-case of "container" than containers as
a special-case of the filesystem. There are containers which can
respond to specifics of the HTTP protocol that are not derived from
files; would you call that part of a "VFS"? The semantics of
"__builtins__.open" are not sufficiently rich to support that.
> In my search for Python VFS's I found PyVFS (http://www.pycage.de/).
> This is modelled on the Gnome/MC VFS, so it supports
> '/dir/somefile.tar.gz#tgz:/somepath' style paths to look inside tar,
> tgz, ftp, whatever. The various different VFS backends are loaded
> dynamicly as pluggins. These pluggins execute as a seperate process
> that are communicated with over a channel. The API is too simplistic
> for me, with files being "projected" out of the VFS to a local file to
> be manipulated/used. I don't like the pluggin-process-channel
> architecture either.
Why not? It sounds clever to me. I don't have a good picture of your
requirements (other than "HTTP and FTP" at this point, so I can't fathom
why you like or don't like this particular solution. (Why is it
> In the reading of the Twisted mailing list, I saw a comment to the
> affect that
> the Medusa VFS was an example of how _not_ to do it, which lead to
> using webDAV
> as the API for a VFS. My gut feeling is DAV is a cool protocol for a VFS
> backend, but I dunno about using it as the primary API. Sure, it
> supports meta-
> data etc, but the reality is the API that is most widely used and
> understood is
> the POSIX filesytem API, as exposed in Python by the os and os.path
Yes, but that's a blocking API; ergo, it does not work in an
asynchronous framework like Twisted. Not all "files" are associated
with a file descriptor, so they may potentially support different
operations. Directory listing on FTP, HTTP, and WebDAV sites is not
necessarily consistent with the files that are actually available.
There is metadata associated with some requests, and not with others...
in short, there are lots of subtle issues involved with supporting each
of these types of hierarchies well, and a blanket virtual "filesystem"
implementation does not satisfy all (or even a reasonably large subset)
of them. For different protocols, there may be API differences, unless
some of the protocols are stripped to the "lowest common denominator",
> My solution for a VFS has been, upto now, based on Medusa's, but
> extending it
> to be more like os and os.path. So far it's a filesystem class with
> most of the
> os and os.path methods. One of the derived classes is a
> that allows you to mount other VFS filesystems off it. At this point I'm
> tempted to make a vfs module that emulates os and os.path so that you
> can mount
> whatever vfs's you want first, and then just replace all your os.*
> calls with
> vfs.* calls. Note that the one catch would be open() would need to be
> with vfs.open().
Not necessarily. You could always hack up the __builtins__ module at
runtime to point to your newer, better open(). Either way it seems like
there are probably issues with security & the presence of *real* file
descriptors that you have to think about...
> I'm sort of fishing for general suggestions, comments, and interest.
> I'm at the
> point where I've just convinced myself my vfs is worth finishing, and
> my ready
> () patch to asynchat is worth updating, but I'm not sure what to use as
> http and ftp server front-end, though I'm still leaning towards medusa.
> looks like Twister is not ready, and ZServer would be too hard to
> seperate from
Ready for what, is the question? :-)
Short form: HTTP yes, FTP no (but it could be with a little work),
everything else yes.
We do not currently have (or have any high-priority plans to produce) a
"VFS", but I would contend that such a system is not necessary when you
look at the way that Twisted does web resources. FTP doesn't currently
use that model, but I don't think it would be a difficult modification;
it uses the same Producer model that HTTP does. I estimate that it
would be easier to modify Twisted in this way than to undertake a
project to do your own VFS, but since I'm not exactly sure what's going
to make you happy, I don't have a high degree of confidence in that
As far as robustness (which is an implied issue with Twisted's
"maturity"), Twisted is being used to run the main twistedmatrix.com
site. We always run the most recent development version. So far, it's
only served 17.422 downloads of Twisted itself and approximately three
quarters of a million hits, and survived five or six freshmeat
"attacks" :-). It did crash, once, about 8 months ago after a major
refactoring (before we had acceptance tests...), but that bug was
quickly fixed. We don't run a public FTP server, but the core parts of
Twisted that enable one to run a website where you can download files
are very personally important to me, so draw your own conclusions :-).
______ you are in a maze of twisted little applications, all
| |_\ remarkably consistent.
| | -- glyph lefkowitz, glyph @ twisted matrix . com
More information about the Twisted-Python