[Twisted-Python] Subproject releases

[James Y Knight]

On Oct 12, 2005, at 10:29 PM, Glyph Lefkowitz wrote:
> Hmm.  Trial interaction is something I haven't messed with yet; I
> haven't put trial directory below a __path__ entry.
>
> Arguably this is simply a bug in Trial that should be fixed, though.
>

The main problem here is that trial's "recursively find all test  
modules for me" mode is based upon a recursive walk of the filesystem  
below a certain path. Currently, "trial twisted" does the following:
1) import the module named ("twisted")
2) os.path.walk over all files under the directory of that module's  
__file__ ("twisted/")
3) filter out (a) any directory which doesn't contain an __init__.*,  
and (b) any files that don't match test_*.py.
4) backwards convert the path of the matched files to a module name  
with some poor heuristics (reflect.filenameToModuleName).
5) try to import said module.

This completely breaks down in the case where packages use __path__.  
The new hierarchy has the following:
Root/
Root/twisted/
Root/twisted/__init__.py (has __path__="core/twisted/","web2/twisted/")
core/
core/twisted/
core/twisted/__init__.py
core/twisted/test/
core/twisted/test/__init__.py
core/twisted/test/test_tcp.py
web2/
web2/twisted/
web2/twisted/web2/
web2/twisted/web2/__init__.py
web2/twisted/web2/test/
web2/twisted/web2/test/__init__.py
web2/twisted/web2/test/test_http.py

So, given that layout, currently, "trial twisted" would get  
twisted.__file__ == "Root/twisted/__init__.py", take the dirname,  
"Root/twisted/", and walk that hierarchy looking for tests (and there  
aren't any there. Oops). Okay, let's specify the directory explicitly  
-- I'll say "trial web2/twisted/web2/test/". Now, that doesn't work  
right either, because it's impossible to backwards convert an  
arbitrary filename to the python module name it's expecting to be  
imported under. filenameToModuleName in this case would return  
"web2.test.test_http", which is wrong.

So, my solution:
I changed the recursive process to actually import every package (not  
every module!), so it can use the package's __path__ attribute to  
find submodules. This works, mostly.

One issue is that it causes trial to emit an import error message  
from trying to import twisted.internet.iocpreactor and  
twisted.internet.serialport (because they aren't importable on my  
system). This is necessary -- trial does not know they don't have a  
__path__ attribute before importing them, so it has no way of telling  
that they don't have test modules hidden inside. It would similarly  
be a bad idea to simply ignore all import errors of packages, as then  
real errors may be silenced. This could be simply solvable via a  
special attribute like "no_tests_in_packages= ['iocpreactor',  
'serialport']" to twisted/internet/__init__ so trial knows not to  
bother trying to import those.

The other issue is that test files expecting to be imported as  
modules _still_ won't work if specified by filename. But, I think  
that's just a fact of life -- it is not possible in general to do the  
correct back conversion. AFAICT we will just have to accept this  
slight functionality loss and recommend that everybody specify module  
names rather than file names (at least for running twisted's tests --  
it's only broken when filenameToModuleName is wrong). Also, Trial  
*does* run the file specified even if it can't figure out how to  
import it, so if the test code doesn't actually depend on having its  
real module name (e.g. it doesn't use things like relative imports or  
__name__), it will also still work.

James