Tests which require python2.5 syntax are inconsistently skipped.

[glyph]

The tests in twisted.test.generator_failure_tests are defined in a regular python file, but conditionally imported from a test_ module. The tests in twisted.test.test_defgen, on the other hand, are defined in a giant string. The former form is easier to work with, modulo some packaging issues such as #1696. We should change the tests to be dealt with in a consistent way. I would prefer the way that avoids 'exec' statements.

[exarkun]

Did you mean the former is easier to work with?

[glyph]

Replying to exarkun:

Did you mean the former is easier to work with?

Oops, yes. And just in case I'm confused again: I want to get rid of the exec and the giant string, and use the conditional-import variety.

[Screwtape]

As the author of the current patch for #1696, I see the problem this way:

• Twisted sometimes adds support for new Python syntax features before it abandons support for older language features. In particular, it has adopted features that require Python 2.5's generator extensions while Python 2.4 is still a supported platform.
• Maintaining compatibility is a matter of hiding code using the new syntax from old versions of Python.
• There's (at least) three different ways such code might be found and fed to the Python interpreter:
  • An application might import an module defining an API that requires the new syntax.
  • A test runner might sniff out tests of the new functionality and run them.
  • An installer might try to precompile all the Python files it can find during installation.
• Application authors who explicitly import and use APIs requiring newer Python versions presumably know what they're doing; godspeed and good luck.
• Test runners generally only look at filenames matching a particular pattern when hunting for tests; the offending syntax can be put into a file with a non-matching name and conditionally imported into a file with a matching name.
• The RPM installer in particular will try to precompile every .py file it can find, whether it's imported or not, and choke on code that requires newer Python versions.

Some solutions (some stupid, some potentially less so) include:

• Bump Twisted's Python requirements whenever any API requires it, even an optional one.
• Put the offending code in strings that are conditionally exec'd at runtime.
• Put the offending code in files that do not match *.py and read them with execfile.
• Officially un-support packaging systems such as RPM that pre-compile Python source files.
• Add an authoritative list of source files that require a Python version higher than the base Twisted requiriment, and exactly which Python version they require. Installers building a distribution of Twisted for a particular version of Python would be required to filter out any files that would not work with that version.

This ticket is definitely a blocker for #1696.

[exarkun]

Add an authoritative list of source files that require a Python version higher than the base Twisted requiriment, and exactly which Python version they require. Installers building a distribution of Twisted for a particular version of Python would be required to filter out any files that would not work with that version.

Of the ideas given, this seems like the most attractive to me. Maintaining such a list shouldn't be too difficult for us, since there are a very small number of files that fall into this funny case, and the supported Python versions don't change too often. Potentially there's even the possibility to automatically generate such a listing as part of the release process from a more convenient definition (like markers in the files that have special requirements).

How feasible is it for distro packages to take advantage of such information, though? Based on past experience, if we don't make it pretty easy (which usually means being just like every other upstream package) it's not likely that the distro package will actually be made to behave well, even if it's technically possible. Since we (upstream) is probably more likely to invest effort in making this work, we should probably let the distro packaging requirements (in particular, ease of) guide our decisions, even if it means making more work for us (upstream).

[glyph]

(Don't mind me, just updating the description based on what exarkun noted to me 2 years ago.)

[itamar]

Note that since we're using modern versions of Python now, we no longer skip any inlineCallbacks tests. This ticket seems worth keeping around, maybe, for the general discussion of what policy should be when dealing with such cases in the future.