Deprecate pickling support (__getstate__ and __setstate__) in twisted.internet._sslverfiy

[itamar]

Pickleability is no longer a goal for Twisted. As such, the various __getstate__ and __setstate__ methods of twisted.internet._sslverify classes should be deprecated, so we can remove the extra code involved.

[itamar]

See twisted/python/reflect.py and twisted/test/test_reflect.py for an example of deprecation and testing of deprecation.

[canibanoglu]

Deprecation of getstate and setstate along with new test which check for deprecation

[tom.prince]

The patch you attached is reversed, and appears to be against the most recent release, rather than trunk. If you are planning on contributing to twisted, you should be working in a source checkout (see [TwistedDevelopment]).

[tom.prince]

The attached patch appears to not apply to trunk.

1. The deprecations want to use twisted.python.deprecate.deprecated, rather than calling warnings.warn directly. The message should probably say something about pickling not being supported anymore.
2. Running

   python -Wall bin/trial twisted.test.test_sslverify
   

   gives me a bunch of deprecation warnings. They should be suppressed.

[canibanoglu]

Thanks a lot for the input.

First of all, the release and trunk version mixup was bordering on idiocy, sorry about that.

1. I have tried using twisted.python.deprecate.deprecated but it doesn't pass the tests because it doesn't work the same way for class methods and module-level functions. I found a ticket which is now marked as fixed but deprecated still acts in a peculiar way. More specifically this is what I get

{{{twisted.trial.unittest.FailTest: not equal: a = 'twisted.internet._sslverify.OpenSSLCertificateOptions.getstate was deprecated in Twisted 13.1.0' b = 'twisted.internet._sslverify.getstate was deprecated in Twisted 13.1.0'}}}

2. Thank you very much for pointing that out, I will definitely take care of that.

[canibanoglu]

New patch on trunk which deprecates getstate and setstate in twisted.internet._sslverify and adds a 6166.removal in twisted/topfiles

[canibanoglu]

Here's my second go at it.

• I have suppressed deprecation warnings when running the tests with

  python -Wall bin/trial twisted.test.test_sslverify
  

• Still can't get @deprecated to work correctly with class methods. This patch still uses warnings.warn for deprecation warnings. Added "Pickling is no longer supported." to the deprecation warning message.

• Added 6166.removal topfile.

I have a question though. While working on this I have found that @deprecated decorator is not functioning as expected when decorating class methods and the full test run hangs on OS X Mountain Lion (10.8). There are tickets for both of these and they both appear as fixed. Should I create another ticket?

Thanks a lot for the assistance.

[exarkun]

New patch on trunk which ...

Can you re-attach the patch, this time without a space in the filename? trac cannot serve attachments with spaces in them.

[canibanoglu]

There's an underscore between 6166 and 2 :) Should I rename them differently from now on?

[exarkun]

Sorry. I guess there's also a problem with filenames with "#" in them.

[canibanoglu]

Removed # from the filename

[canibanoglu]

Uploaded the same patch file without the "#" character in the filename.

[tom.prince]

1. There are heplers to suppres warnings. See twisted.trial.util.suppress and the .suppress attribute of TestCase and/or test_* methods.
2. Hmm. Looking at the r24931 diff for #3254, deprecated works with methods, but it needs to called as

   Class.method = deprecated(version)(Class.method)
   

   after the class definition. You should file a ticket to support the obvious thing.
   • (Making this work is probably tricky, and would involve making deprecate return a descriptor)
3. Is there any reason you created your own helper, rather than using callDeprecated?
4. The newsfile could be improved by listing all the things that are getting deprecated, rather than saying each individual thing is deprecated, but might be better to just say that pickling the two relevant classes is now deprecated.
5. Even if not using @deprecated, you should still use the same format for the docstring.

[canibanoglu]

Tests now use callDeprecated to test deprecations, use .suppress attribute to suppress deprecation warning messages

[tomprince]

(In [38505]) Branching to sslverify-no-pickle-6166.

[tom.prince]

This looks good. A couple of points:

1. The suppression for test_certificateOptionsSerialization is to general.
2. CertificateOptions and KeyPair both live publicly in `twisted.inte

rnet.ssl`, so the news should reflect that.

[tomprince]

(In [38507]) Address review comments.

Refs #6166.

[tomprince]

(In [38518]) Be even more specific about what to suppress.

Refs #6166.

[tom.prince]

To expand one (1), we only want to suppress the specific warnings that we are triggering by testing this code. That way, if somebody later changes something else to cause a different warning to occur here, it would still be reported.

Other than those two points, this looks good.

I'll go ahead and merge after fixing those two issues.

[tomprince]

(In [38519]) Merge sslverify-no-pickle-6166: Deprecate pickling support in twisted.internet._sslverfiy.

Authore: canibanoglu Reviewers: tom.prince Fixes: #6166

Pickleability is no longer a goal for Twisted. As such, the various getstate and setstate methods of twisted.internet._sslverify classes can be deprecated, so we can remove the extra code involved.

[exarkun]

(In [38527]) Revert r38517 - test suite regression

Reopens: #6166

A number of test modules now fail to import on some builders:

[ERROR]
Traceback (most recent call last):
  File "/data/home/buildbot/work/freebsd-8.2-i386/Twisted/twisted/trial/runner.py", line 500, in loadPackage
    module = modinfo.load()
  File "/data/home/buildbot/work/freebsd-8.2-i386/Twisted/twisted/python/modules.py", line 383, in load
    return self.pathEntry.pythonPath.moduleLoader(self.name)
  File "/data/home/buildbot/work/freebsd-8.2-i386/Twisted/twisted/python/_reflectpy3.py", line 266, in namedAny
    topLevelPackage = _importAndCheckStack(trialname)
  File "/data/home/buildbot/work/freebsd-8.2-i386/Twisted/twisted/python/_reflectpy3.py", line 205, in _importAndCheckStack
    return __import__(importName)
  File "/data/home/buildbot/work/freebsd-8.2-i386/Twisted/twisted/test/test_sslverify.py", line 805, in <module>
    class KeyPair(unittest.TestCase):
  File "/data/home/buildbot/work/freebsd-8.2-i386/Twisted/twisted/test/test_sslverify.py", line 808, in KeyPair
    CN=b"server")[0]
  File "/data/home/buildbot/work/freebsd-8.2-i386/Twisted/twisted/test/test_sslverify.py", line 84, in makeCertificate
    keypair = PKey()
exceptions.NameError: global name 'PKey' is not defined

twisted.test.test_sslverify

[exarkun]

(In [38528]) whitespace fixes

refs #6166

[tomprince]

(In [38537]) Branching to sslverify-no-pickle-6166-2.

[tom.prince]

I think that should address the test failures.

[glyph]

Minor nitpick: ​there is a new twisted checker error; a new line is too long.

[tomprince]

(In [38926]) Wrap long line.

Refs: #6166

[habnabit]

There was no buildbot run for this, so I'm forcing one now.

[rwall]

Notes:

• Merges cleanly

Points:

1. Build Failures
   1. Python3 tests are very unhappy: ​https://buildbot.twistedmatrix.com/builders/python-3.3-tests/builds/1405/steps/shell/logs/stdio
   2. The circularChainWarning failure is something that has since been fixed in trunk I think (r39074)
   3. The output of python3 tests seems unusual. It only mentions failed and skipped tests, not the ones that passed.
   4. It also says it dropped to a debugger which I don't remember seeing before.
   5. So try merging forward? Maybe that'll fix all the skips too.
2. Deprecation version numbers need updating. 13.1.0 > 13.2.0
3. source:branches/sslverify-no-pickle-6166-2/twisted/test/test_sslverify.py
   1. "util.suppress(category = DeprecationWarning" -- remove whitespace in the argument assignment.
   2. "Test deprecation" -- remove the "Test" from test docstrings.
   3. "test_setstateDeprecation.suppress" --
      1. Do you have to pass a full state dictionary in the setstat test?
      2. If you could just pass an empty dict (or minimal dummy dict) this extra suppression might not be necessary.
      3. The wrapping and message formatting looks strange -- don't the quoted strings need to be surrounded in round brackets?
      4. Fix the whitespace in the argument assignment.
   4. "KeyPair" many of the same comments apply to this new testcase.
4. source:branches/sslverify-no-pickle-6166-2/twisted/internet/_sslverify.py
   1. Can you move the deprecation decorators directly beneath the affected function, to make it obvious that it's deprecated.

Please, merge forward, address or answer the numbered points above and link to clean build results then merge.

Thanks.

-RichardW.

[tomprince]

(In [42225]) Branching to sslverify-no-pickle-6166-3.

[tom.prince]

The decorators have to be after the class definition, unfortunately.

[exarkun]

Thanks.

1. There's a twistedchecker error in _sslverify.py that it might be nice to fix here.
2. The suppression is defined as a regular expression. This code takes advantage of that fact in some ways but ignores it in others - for example, it doesn't escape the "." module separator.
3. It's unfortunate that 14.0 hasn't been released yet. I think that 14.1 is the correct version for this deprecation. It wouldn't surprise me if some people try to restart the 14.0 release process based on a new trunk revision though, that would cause a problem here. I don't know what to do about that except blame those people for screwing things up if that happens, though.

Please address those points and then merge. Thanks.

[tomprince]

(In [42375]) Branching to sslverify-no-pickle-6166-4.

[tomprince]

(In [42380]) Merge sslverify-no-pickle-6166-4: Deprecate pickling support in twisted.internet._sslverfiy.

Author: canibanoglu, tom.prince Reviewer: exarkun, tom.prince, rwall Fixes: #6166

Pickling twisted.internet.ssl.OptionSSLCertificationOptions and twisted.internet.ssl.Keypair is no longer supported.