how winnt fileops work and what to do about it (was Re: [Twisted-Python] Twisted windows hackers - help the tests to pass!)
paul-lists at perforge.com
Sat Dec 31 05:48:43 EST 2005
ok, i can actually chime in here because i've done filesystems work on
windows (don't ask ;). now, it's been a while, but i should remember things
reasonably accurately (i hope). see below for comments:
----- Original Message -----
To: twisted-python at twistedmatrix.com
Sent: Saturday, December 31, 2005 2:15 AM
Subject: Re: [Twisted-Python] Twisted windows hackers - help the tests to
> It's not so much a misconfiguration, as an issue with the fact that trial
> on Window s is failing to rename its _trial_temp folder,
> because there are files inside it that are open. I've opened a bug on it
> over on < http://twistedmatrix.com/bugs/issue1387>.
it is incorrect to state that file deletion will always fail when the file
is open. let me explain how nt/ntfs/win32 does this, because it's highly
'strange' to folks from a posix background. this may be way more info than
you were looking for, but i'm going to type it out anyway so that someone
can then make the call on how to deal with the issue.
for functionality implemented in the kernel, such as ntfs, windows has 2 api
layers: win32 (you see CreateFile() here) and the native nt api. win32 is
implemented in usermode, while the native api is implemented as a kernel
service, which is exposed to userspace code with the Zw prefix (ie
ZwCreateFile()). sometimes the win32 api calls map directly, sometimes they
are multiplexed (ie one win32 api call may use multiple native api calls to
do the work). what happens here is undocumented (at least officially and
without an nda). what is important is that some calls allow you to do things
which win32 does not expose; most popular ones deal with io completion (this
is how you cancel async i/o on windows nt).
now, on to the meat: the win32 CreateFile() is actually a jack-of-all-trades
call (horrible design), used for file creation, opening *and authorization*.
we're not interested in ACLs in this case, but we are interested in locks -
locks in nt are compulsory (as opposed to purely advisory in posix) and
violating one will result in an auth failure. now, disregarding the ability
to take out range locks (locks on a byterange within a file), CreateFile()
takes out certain locks based on the sharemode (iirc, not sure) parameter.
the rules here are funky, iirc, but the one we care about is
FILE_SHARE_DELETE. *unless* this flag is set in sharemode passed to
CreateFile(), all attempts to open this file for deletion (see,
authorization) going forward will fail.
it will fail when the file is opened exclusively. CreateFile takes a
parameter called sharemode, iirc, which can be a combination of
FILE_SHARE_READ, FILE_SHARE_WRITE, and FILE_SHARE_DELETE. *unless*
FILE_SHARE_DELETE is set in the flag parameter, a lock preventing deletion
gets taken out and you get to enjoy all the wonders of compulsory locking
you are hitting.
here's why i think this happens: win32 DeleteFile() does *not* actually use
ZwDeleteFile() native call to delete a file. instead, it does a CreateFile()
(or ZwCreateFile(), could be either) to open the file (and get a handle to
it), telling it the desired access is DELETE. then it does a
ZwSetFileInformation(), which it tells it wants to set 'disposition' and
passes in the appropriate disposition info struct with the DELETE
disposition set. you can't cheat this, because ZwSetInformation() will fail
if the handle doesn't have DELETE rights, and you won't get them if the file
isn't opened with FILE_SHARE_DELETE. bummer. the reason this is all so
roundabout is that the file whose disposition is set to DELETE doesn't
actually get deleted until the last handle is ZwClose()'d - this is where
the deletion takes place. now, this is from memory, so don't hold me to it
exactly, but the jist of it should be correct.
the interesting, albeit non-obvious, question is: what does ZwDeleteFile()
do? it takes either a handle *or a path* and deletes it *right away*,
without waiting for handles to be closed. now, i don't know whether it
bypasses the ZwCreateFile() and hence the DELETE check, but there's a chance
that it is the call ZwClose() makes internally when it does a delete based
on the disposition (this guess would be supported by the fact that
ZwCreateFile() and a few other fs calls are documented in msdn/ddk, but
ZwDeleteFile() is not) and hence doesn't hit this check. what is even more
interesting is that, in win32, the only way to remove a directory is using
RemoveDirectory(), which requires the directory to be empty. you have to
recursively delete all contents, either by hand or using SHFileOperation,
iirc. this is the op that hits your open files problem. i remember seeing
code (this part i didn't work on, but did read brielfy) which deleted
directories with ZwDeleteFile() and there was no resurive content deletion
code, so i suspect that you could delete a non-empty directory this way.
neither of this is valid ntfs usage, so doing it may not be kosher.
whatever the case may be, those two options are available if you can stomach
> going to need a rethink on how _trial_temp works, because my instant
> thought on how to solve was "symlinks" and python
> doesn't support them on windows, mostly because windows' own support of
> them is a tad on the "dont' ask, don't tell" side of
> things, and NTFS-only anyway.
NTFS and DFS, but yeah, no FAT (if anyone cares). symlinks in ntfs can be
implemented using what's called 'reparse points'. these are actually quite
powerful - you can attach either a static transformation or code (you need a
driver for this, iirc) to a certain dentry these symlinks are called
'junctions', but they work only for directories. currently, all of this is
extremely hairy to use. there are also hardlinks, which you can actually
create with the win32 api, but they are only for files moreover, none of
this behaves like symlinks and hardlinks in terms of finer semantics (ie
unlinking). junctions are most closely related to mount --bind, rather than
symlinks, for example.
with all of the above said, you've basically got these choices (in no
particular order) to deal with the issue at hand:
1. change the offending code not to do this rename
2. instead of a rename, create a new directory, do a recursive copy into it
from the original and retry removing the original directory asynchronously
until it succeeds. obviously, the file handles which are open at that point
will be referencing a different copy of the files from the ones which will
be opened subsequently.
3. test whether the zwDeleteFile() behaves in the way i conjrectured it to
(wrt files or directories). if so, cause it to be implemented in the pywin32
extension and use it to perform the delete.
4. test whether you can either use ZwSetFileInformation() to rename
directories by changing the FILE_NAME attr in the appropriate info structure
or use it to move by renaming files which are open, again using the
appropriate (but different) structure.
if so, implement this or cause this to be implemented in pywin32. it is
unclear (to me) whether this would result in the pre-move file handles being
dead, stale or correct.
5. instead of opening the files as normal, open them with pywin32's
implementation of CreateFile(), specifying the appropriate sharemode. this
will allow the rename (move really) to go through, but it is unclear what
happens to the preexisting filehandles.
6. implement, or cause to be implemented, CreateHardlink() in pywin32.
create a new directory and recursively hardlink contents of the original
into the new directory. asynchronously retry recursive deletion of the
that's all i can think of anyway.
>MFen tracked down an error involving "r+b" mode. Seems windows handling of
>it is insane. See ><http://twistedmatrix.com/bugs/issue1386>. This could
>well be a python bug, or a feature of windows.
i'm not sure how python does file opens and i/o on windows. with that said,
assuming that it uses fopen() from the visual studio c runtime library,
there is a quirk in the implementation that might be causing this. if you
use any of the + modes, ie a+, r+ or w+. when you switch between reading and
writing you need to do an fflush() or fsetpos() (possibly some others like
fseek() could work too, don't remember). try doing a file.flush() on your
file object somewhere in there and see if that fixes things for you. if it
does, this should probably be reported as a python stdlib bug.
it's really late, so pardon the wordiness and possible inaccuracies due to
More information about the Twisted-Python