[Twisted-Python] Re: Twisted 1.0.4 - Heavy search for `plugins.tml'

Francois Pinard pinard at iro.umontreal.ca
Sun Apr 27 20:28:48 EDT 2003


[Christopher Armstrong]

> > Is Twisted modifying `sys.path' to include the first part of argv[0]?
> > Or is it standard Python behaviour?

> Yes, Python always adds the location of the main script to sys.path [...]

Thanks for enlightening me.

For the case of the heaviness for `plugins.tml', at least on Unix, there
might be something we could do that would be reasonable on average,
but I'm not fully sure, I'll merely let judge those who are grown up! :-)

Usually, `/usr/bin/' and such do not have subdirectories, so if a
directory contains 1000 entries, one may immediately check if one of
these is `plugins.tml' by a direct try.  Then, if the number of links of
that directory -- itself -- is exactly 2, we can conclude that it has no
sub-directories, and that there is nothing more to check for this one.
We could then get more speed even for directories other than `/usr/bin/'.

This trick is heavily used in GNU find as a way to avoid `stat'ing all
entries in a directory to find sub-directories, when it can be proven
in advance there is no such sub-directories.  When the trick was added,
the speed-up that resulted was judged spectacular.

But I do not remember if Linux existed at the time.  As Linux seems to
beat many other systems at properly caching disk accesses, the benefits
of the trick might be more hidden.  A year ago, maybe, I tried the same
trick on Linux in hope of increasing the performance of `os.path.walk',
and the speed-up was not so significant.  This is part of my hesitation.

Moreover, maybe the above trick is meaningless on Windows or on MacOS.
I would be surprised if there are Unix filesystems where the link count of
2 for directories (`/' excepted) is not dependable; this might be another
thing to check.  With some luck, GNU find sources would tell us what the
problems may be.  The maintainers surely met them all! :-)

-- 
François Pinard   http://www.iro.umontreal.ca/~pinard




More information about the Twisted-Python mailing list