[Twisted-Python] Bloody Twisted Tree (VFS)
jml at mumak.net
Sun Jul 6 19:26:00 EDT 2008
On Mon, Jul 7, 2008 at 1:12 AM, <glyph at divmod.com> wrote:
> On 07:32 am, jml at mumak.net wrote:
>> On Sat, Jul 5, 2008 at 3:27 AM, Andy Gayton <andy at thecablelounge.com>
>>> On Mon, Jun 30, 2008 at 9:06 PM, Jonathan Lange <jml at mumak.net> wrote:
>>> * The primitive interface for IO should be producer/consumers,
>>> replacing readChunk, writeChunk. This interface is primitive enough
>>> to express all other interfaces, while still providing the opportunity
>>> to optimize streaming performance. The producer/consumer interface
>>> will need to take an offset to allow readChunk and writeChunk to be
>> It would be nice to have things so that readChunks and writeChunks
>> (plural) could be implemented, to avoid potato programming.
> I don't think this is actually going to be a practical consideration, if I
> correctly understand what you mean. For one thing, the producer/consumer
> interface is going to be something (very vaguely) like this:
> remoteFile.writeFrom(producer[, offset, length])
> remoteFile.readInto(consumer[, offset, length])
> This means that if you've got a really giant file, the implementation could
> pretty trivially optimize delivering it to you in the most efficient
> possible way, keeping all the relevant buffers full at every opportunity.
> Given that stream-based I/O is somewhat inherently serial, it's difficult
> to get less potato-y than that.
> writeChunks, if I understand it, would be pretty trivially implementable by
> Mapping 'readChunks' and 'writeChunks' to readv and writev in my head, I'm
> not really sure what a 'readChunks' would actually do, since we copy memory
> every time we sneeze in Python anyway. We're not going to have preallocated
> buffers to read into.
Well, one theoretical advantage is that it can avoid roundtrips in
cases where the remote file server supports a readv-style operation. I
can't think of any servers that do this at the moment (maybe the bzr
smart server? does http 1.1 allow this?), so maybe it's not an issue.
>> This reminds me, it would be good for VFS to have an exception for
>> "this operation isn't supported" (say with symlinks on fat32) and
>> another exception for "supportable, but not actually implemented yet".
> I don't think it's useful to distinguish between these two types of
> exception at *runtime*. The use-case I can see for distinguishing is
> letting a programmer know that they should figure out something that might
> be tricky to implement and write some wrappers or submit some patches.
> Perhaps a separate error message, rather than a separate exception type?
> Do you have a different use-case?
The first kind should skip tests, the second kind should fail tests.
> One related thing that we spoke about in person was pushing this negotiation
> of file-system features backwards to the initialization step, so that
> applications which needed unusual filesystem attributes could fail quickly
> with a clear error message if they weren't supported by the underlying
This sounds like a good idea, provided that there are still clear
runtime errors and that you can skip the negotiation.
Use cases for this would be a virtual filesystem that's glued together
from other virtual filesystems, each of which has different
> ("WebDAV requires extended filesystem attributes, and your
> backend, SFTP, does not provide that feature.", "txGnuStow requires symbolic
Actually, some versions of SFTP do provide it. I'm not sure that there
are any implementations though :)
>>> * we still need to decide whether path resolution should be moved to
>>> a separate interface, instead of being part of the node's interface.
>> I'm not 100% sure what this means? Does this relate to possibly
>> combining with FilePath?
> The tongue-in-cheek name that radix gave to this interface was
> 'filepath.pathdelta'. It's related to filepath in the sense that FilePath,
> ZipPath, et. al. could benefit from using the same interface to talk about
> relative pathnames rather than manipulating lists of strings. One can,
> after all, abstractly do operations like "child()" and "parent()" without
> knowing a lot about the base implementation of the filesystem in question.
Well, the world needs a decent one of these.
>>> * there's concern over the package name. twisted.tree has
>>> considerable support :)
>> I kind of like that. I'm not sure what the concern is with 'vfs' though.
> "twisted.vfs" sounds incredibly boring and unpronounceable. It would be the
> first twisted.<acronym> package, and it's not really related to any other
> technology ambiguously named "vfs".
> However, this reminds me about another concern which I did not remember to
> raise while Andy was here. Should this really be twisted.<anything> at all?
> I'd like twisted <x> "dot products" to generally be an application which
> does something <x>-ish. I'm aware that not every package follows this rule,
> but the ones that don't are either (A) unmaintained and slated for removal,
> or (B) part of the core, not independent subprojects, as "vfs" seems slated
> to be.
> Put a different way: what should 'twistd tree' do? My suggestion would be a
> simple multi-protocol file server: HTTP, FTP (although probably disable that
> by default), SFTP, maybe a "native" protocol for providing a generalized
> backend for any Twisted application that uses the 'tree' API, so that we can
> write a proxy that exposes every arbitrary combination of features from the
> protocols it's talking to.
> If everyone agrees with this, then great. However, if we never intend for
> this to go beyond providing an API that other systems hook into, maybe it
> should go somewhere subordinate to another project; twisted.internet.files
So, this is the thing that *I'm* least worried about.
I think it should just be an API, and that it should be done so that
other Twisted components can depend on it. Beyond that, it's package
location is unimportant.
>> Here's some random stuff that I wanted to at least mention:
>> - Error translation. This should translate the exception types, but it
>> should also translate values, so the error contains the virtual path.
> This sounds like a specific enough thing that you could file a ticket that
> described the exact behavior that you wanted. It doesn't sound contentious
> at all to me, so unless you think there's some hidden confusion there... go
>> - Deferreds. You don't mention them at all, but the lack of
>> asynchronous interfaces was one of the biggest problems we had with
> I believe that the consensus on asynchronicity is that all of the
> synchronous stuff should be FilePath's job. In the glorious future of
> twisted.tree, everything will be async. As discussed above, this doesn't
> always mean Deferreds, it also means producers and consumers.
> One thing we didn't talk about in person: handling extremely large
> directories. We had spoken about children() returning a Deferred of a list;
> I think it would be nice if it actually had a producer/consumer API of its
> own. Maybe this is too much of a corner case to worry about in average
> applications (i.e. we could provide a give-me-a-deferred convenience API)
> but it would be nice if it were *possible* to implement things that were
> efficient against really big networked directories.
Yes. This would be very nice.
>> - URL Escaping. I got bitten by this recently. It's obviously not a
>> general VFS problem, but it's an issue with enough of them that it
>> should be considered when defining interfaces.
> I *think* that this should be pretty easily dealt with in a pretty generic
> way by having a clearly-defined set of string escaping rules depending on
> which protocol you're using. It's a general VFS issue in the sense that
> there are escaping issues with "/" on regular filesystems, after all. Or at
> least, there are error-reporting issues with characters like "/", ";", and
> ":" on certain FSes.
Good. I just wanted to flag it.
More information about the Twisted-Python