[Twisted-Python] unified filesystem API

[Glyph]

> On May 15, 2020, at 11:30 PM, Chris Withers <chris at withers.org> wrote:
> 
> On 16/05/2020 06:55, Glyph wrote:
>> This does point out one of my secret hopes for SMB: that a file server's maintainers will care enough about file throughput that we'll finally get a centralized, official way of doing async file I/O that we can share with SFTP, FTP, and HTTP :).
> 
> I got excited by the title, but this all seems to be able giving non-filesystems a file-system like API.

What are you referring to as non-filesystems?

> If I've missed something, please let me know, but I couldn't find an async interface for writing to actual file systems.

In Twisted? There isn't one, really, which is exactly the issue.  There are various operating system interfaces for this but none of them are great.

> It feels like Twisted has just punted on them as "fast enough" to not need async interfaces, but that can really not be the case for networked filesystems (NFS, GPFS, etc) where the blocking time for reading or writing can be seconds (or minutes if it's having a bad day!).

Indeed, I'm well aware!  That's what I'm talking about.  It's not quite worth the energy to do it just for your average HTTP server, but it sure would be nice if we did have it, since it works so much better in the edge cases you enumerated.  (Or even against a spinning rust platter under significant load.)

> What's the Twisted solution for these kinds of things? Defer all file IO into a thread?

That is the only really portable solution, but there are sometimes platform-specific ones which can do better in some circumstances.  However, if we had a unified API for accessing the filesystem that presented a non-blocking interface without *talking* about threads, we'd be able to take user code written against that API and instead make it do something smarter than just using a thread - or even just use an intelligent number of threads (one per platter, for example, which you can figure out by looking at various files in /etc, on some platforms, sometimes).

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20200515/fbf146d9/attachment.htm>