Opened 2 years ago

Closed 6 months ago

#6166 enhancement closed fixed (fixed)

Deprecate pickling support (__getstate__ and __setstate__) in twisted.internet._sslverfiy

Reported by: itamar Owned by: tom.prince
Priority: low Milestone:
Component: core Keywords: easy gsoc
Cc: thijs Branch: branches/sslverify-no-pickle-6166-4
(diff, github, buildbot, log)
Author: canibanoglu, tomprince Launchpad Bug:

Description

Pickleability is no longer a goal for Twisted. As such, the various __getstate__ and __setstate__ methods of twisted.internet._sslverify classes should be deprecated, so we can remove the extra code involved.

Attachments (4)

ticket6166.patch (12.1 KB) - added by canibanoglu 19 months ago.
Deprecation of getstate and setstate along with new test which check for deprecation
ticket#6166_2.patch (9.3 KB) - added by canibanoglu 18 months ago.
New patch on trunk which deprecates getstate and setstate in twisted.internet._sslverify and adds a 6166.removal in twisted/topfiles
ticket6166_2.patch (9.3 KB) - added by canibanoglu 18 months ago.
Removed # from the filename
ticket6166-3.patch (5.4 KB) - added by canibanoglu 18 months ago.
Tests now use callDeprecated to test deprecations, use .suppress attribute to suppress deprecation warning messages

Download all attachments as: .zip

Change History (40)

comment:1 Changed 21 months ago by thijs

  • Cc thijs added

comment:2 Changed 19 months ago by itamar

  • Keywords easy added

See twisted/python/reflect.py and twisted/test/test_reflect.py for an example of deprecation and testing of deprecation.

Changed 19 months ago by canibanoglu

Deprecation of getstate and setstate along with new test which check for deprecation

comment:3 Changed 19 months ago by canibanoglu

  • Keywords review added

comment:4 Changed 19 months ago by tom.prince

The patch you attached is reversed, and appears to be against the most recent release, rather than trunk. If you are planning on contributing to twisted, you should be working in a source checkout (see [TwistedDevelopment]).

comment:5 Changed 19 months ago by tom.prince

  • Keywords review removed
  • Owner set to canibanoglu

The attached patch appears to not apply to trunk.

  1. The deprecations want to use twisted.python.deprecate.deprecated, rather than calling warnings.warn directly. The message should probably say something about pickling not being supported anymore.
  2. Running
    python -Wall bin/trial twisted.test.test_sslverify
    
    gives me a bunch of deprecation warnings. They should be suppressed.

comment:6 Changed 18 months ago by canibanoglu

Thanks a lot for the input.

First of all, the release and trunk version mixup was bordering on idiocy, sorry about that.

  1. I have tried using twisted.python.deprecate.deprecated but it doesn't pass the tests because it doesn't work the same way for class methods and module-level functions. I found a ticket which is now marked as fixed but deprecated still acts in a peculiar way. More specifically this is what I get

{{{twisted.trial.unittest.FailTest: not equal:
a = 'twisted.internet._sslverify.OpenSSLCertificateOptions.getstate was deprecated in Twisted 13.1.0'
b = 'twisted.internet._sslverify.getstate was deprecated in Twisted 13.1.0'}}}

  1. Thank you very much for pointing that out, I will definitely take care of that.

Changed 18 months ago by canibanoglu

New patch on trunk which deprecates getstate and setstate in twisted.internet._sslverify and adds a 6166.removal in twisted/topfiles

comment:7 Changed 18 months ago by canibanoglu

  • Keywords review added
  • Owner canibanoglu deleted

Here's my second go at it.

  • I have suppressed deprecation warnings when running the tests with
    python -Wall bin/trial twisted.test.test_sslverify
    
  • Still can't get @deprecated to work correctly with class methods. This patch still uses warnings.warn for deprecation warnings. Added "Pickling is no longer supported." to the deprecation warning message.
  • Added 6166.removal topfile.

I have a question though. While working on this I have found that @deprecated decorator is not functioning as expected when decorating class methods and the full test run hangs on OS X Mountain Lion (10.8). There are tickets for both of these and they both appear as fixed. Should I create another ticket?

Thanks a lot for the assistance.

comment:8 Changed 18 months ago by exarkun

New patch on trunk which ...

Can you re-attach the patch, this time without a space in the filename? trac cannot serve attachments with spaces in them.

comment:9 Changed 18 months ago by canibanoglu

There's an underscore between 6166 and 2 :) Should I rename them differently from now on?

comment:10 Changed 18 months ago by exarkun

Sorry. I guess there's also a problem with filenames with "#" in them.

Changed 18 months ago by canibanoglu

Removed # from the filename

comment:11 Changed 18 months ago by canibanoglu

Uploaded the same patch file without the "#" character in the filename.

comment:12 Changed 18 months ago by tom.prince

  • Keywords gsoc added; review removed
  • Owner set to canibanoglu
  1. There are heplers to suppres warnings. See twisted.trial.util.suppress and the .suppress attribute of TestCase and/or test_* methods.
  2. Hmm. Looking at the r24931 diff for #3254, deprecated works with methods, but it needs to called as
    Class.method = deprecated(version)(Class.method)
    
    after the class definition. You should file a ticket to support the obvious thing.
    • (Making this work is probably tricky, and would involve making deprecate return a descriptor)
  3. Is there any reason you created your own helper, rather than using callDeprecated?
  4. The newsfile could be improved by listing all the things that are getting deprecated, rather than saying each individual thing is deprecated, but might be better to just say that pickling the two relevant classes is now deprecated.
  5. Even if not using @deprecated, you should still use the same format for the docstring.

Changed 18 months ago by canibanoglu

Tests now use callDeprecated to test deprecations, use .suppress attribute to suppress deprecation warning messages

comment:13 Changed 18 months ago by canibanoglu

  • Keywords review added
  • Owner canibanoglu deleted

comment:14 Changed 17 months ago by tomprince

  • Author set to tomprince
  • Branch set to branches/sslverify-no-pickle-6166

(In [38505]) Branching to sslverify-no-pickle-6166.

comment:15 Changed 17 months ago by tom.prince

  • Owner set to tom.prince
  • Status changed from new to assigned

This looks good. A couple of points:

  1. The suppression for test_certificateOptionsSerialization is to general.
  2. CertificateOptions and KeyPair both live publicly in `twisted.inte

rnet.ssl`, so the news should reflect that.

comment:16 Changed 17 months ago by tomprince

(In [38507]) Address review comments.

Refs #6166.

comment:17 Changed 17 months ago by tomprince

(In [38518]) Be even more specific about what to suppress.

Refs #6166.

comment:18 Changed 17 months ago by tom.prince

  • Keywords review removed

To expand one (1), we only want to suppress the specific warnings that we are triggering by testing this code. That way, if somebody later changes something else to cause a different warning to occur here, it would still be reported.

Other than those two points, this looks good.

I'll go ahead and merge after fixing those two issues.

comment:19 Changed 17 months ago by tomprince

  • Resolution set to fixed
  • Status changed from assigned to closed

(In [38519]) Merge sslverify-no-pickle-6166: Deprecate pickling support in twisted.internet._sslverfiy.

Authore: canibanoglu
Reviewers: tom.prince
Fixes: #6166

Pickleability is no longer a goal for Twisted. As such, the
various getstate and setstate methods of
twisted.internet._sslverify classes can be deprecated, so we
can remove the extra code involved.

comment:20 Changed 17 months ago by exarkun

  • Resolution fixed deleted
  • Status changed from closed to reopened

(In [38527]) Revert r38517 - test suite regression

Reopens: #6166

A number of test modules now fail to import on some builders:

[ERROR]
Traceback (most recent call last):
  File "/data/home/buildbot/work/freebsd-8.2-i386/Twisted/twisted/trial/runner.py", line 500, in loadPackage
    module = modinfo.load()
  File "/data/home/buildbot/work/freebsd-8.2-i386/Twisted/twisted/python/modules.py", line 383, in load
    return self.pathEntry.pythonPath.moduleLoader(self.name)
  File "/data/home/buildbot/work/freebsd-8.2-i386/Twisted/twisted/python/_reflectpy3.py", line 266, in namedAny
    topLevelPackage = _importAndCheckStack(trialname)
  File "/data/home/buildbot/work/freebsd-8.2-i386/Twisted/twisted/python/_reflectpy3.py", line 205, in _importAndCheckStack
    return __import__(importName)
  File "/data/home/buildbot/work/freebsd-8.2-i386/Twisted/twisted/test/test_sslverify.py", line 805, in <module>
    class KeyPair(unittest.TestCase):
  File "/data/home/buildbot/work/freebsd-8.2-i386/Twisted/twisted/test/test_sslverify.py", line 808, in KeyPair
    CN=b"server")[0]
  File "/data/home/buildbot/work/freebsd-8.2-i386/Twisted/twisted/test/test_sslverify.py", line 84, in makeCertificate
    keypair = PKey()
exceptions.NameError: global name 'PKey' is not defined

twisted.test.test_sslverify

comment:21 Changed 17 months ago by exarkun

(In [38528]) whitespace fixes

refs #6166

comment:22 Changed 17 months ago by tomprince

  • Branch changed from branches/sslverify-no-pickle-6166 to branches/sslverify-no-pickle-6166-2

(In [38537]) Branching to sslverify-no-pickle-6166-2.

comment:23 Changed 17 months ago by tom.prince

  • Author changed from tomprince to canibanoglu
  • Keywords review added

I think that should address the test failures.

comment:24 Changed 17 months ago by tom.prince

  • Owner tom.prince deleted
  • Status changed from reopened to new

comment:25 Changed 17 months ago by glyph

Minor nitpick: there is a new twisted checker error; a new line is too long.

comment:26 Changed 16 months ago by tomprince

(In [38926]) Wrap long line.

Refs: #6166

comment:27 Changed 15 months ago by habnabit

There was no buildbot run for this, so I'm forcing one now.

comment:28 Changed 15 months ago by habnabit

  • Owner set to habnabit

comment:29 Changed 14 months ago by rwall

  • Owner changed from habnabit to rwall
  • Status changed from new to assigned

comment:30 Changed 14 months ago by rwall

  • Keywords review removed
  • Owner changed from rwall to tom.prince
  • Status changed from assigned to new

Notes:

  • Merges cleanly

Points:

  1. Build Failures
    1. Python3 tests are very unhappy: https://buildbot.twistedmatrix.com/builders/python-3.3-tests/builds/1405/steps/shell/logs/stdio
    2. The circularChainWarning failure is something that has since been fixed in trunk I think (r39074)
    3. The output of python3 tests seems unusual. It only mentions failed and skipped tests, not the ones that passed.
    4. It also says it dropped to a debugger which I don't remember seeing before.
    5. So try merging forward? Maybe that'll fix all the skips too.
  2. Deprecation version numbers need updating. 13.1.0 > 13.2.0
  3. source:branches/sslverify-no-pickle-6166-2/twisted/test/test_sslverify.py
    1. "util.suppress(category = DeprecationWarning" -- remove whitespace in the argument assignment.
    2. "Test deprecation" -- remove the "Test" from test docstrings.
    3. "test_setstateDeprecation.suppress" --
      1. Do you have to pass a full state dictionary in the setstat test?
      2. If you could just pass an empty dict (or minimal dummy dict) this extra suppression might not be necessary.
      3. The wrapping and message formatting looks strange -- don't the quoted strings need to be surrounded in round brackets?
      4. Fix the whitespace in the argument assignment.
    4. "KeyPair" many of the same comments apply to this new testcase.
  4. source:branches/sslverify-no-pickle-6166-2/twisted/internet/_sslverify.py
    1. Can you move the deprecation decorators directly beneath the affected function, to make it obvious that it's deprecated.

Please, merge forward, address or answer the numbered points above and
link to clean build results then merge.

Thanks.

-RichardW.

comment:31 Changed 6 months ago by tomprince

  • Author changed from canibanoglu to canibanoglu, tomprince
  • Branch changed from branches/sslverify-no-pickle-6166-2 to branches/sslverify-no-pickle-6166-3

(In [42225]) Branching to sslverify-no-pickle-6166-3.

comment:32 Changed 6 months ago by tom.prince

The decorators have to be after the class definition, unfortunately.

comment:33 Changed 6 months ago by tom.prince

  • Keywords review added
  • Owner tom.prince deleted

comment:34 Changed 6 months ago by exarkun

  • Keywords review removed
  • Owner set to tom.prince

Thanks.

  1. There's a twistedchecker error in _sslverify.py that it might be nice to fix here.
  2. The suppression is defined as a regular expression. This code takes advantage of that fact in some ways but ignores it in others - for example, it doesn't escape the "." module separator.
  3. It's unfortunate that 14.0 hasn't been released yet. I think that 14.1 is the correct version for this deprecation. It wouldn't surprise me if some people try to restart the 14.0 release process based on a new trunk revision though, that would cause a problem here. I don't know what to do about that except blame those people for screwing things up if that happens, though.

Please address those points and then merge. Thanks.

comment:35 Changed 6 months ago by tomprince

  • Branch changed from branches/sslverify-no-pickle-6166-3 to branches/sslverify-no-pickle-6166-4

(In [42375]) Branching to sslverify-no-pickle-6166-4.

comment:36 Changed 6 months ago by tomprince

  • Resolution set to fixed
  • Status changed from new to closed

(In [42380]) Merge sslverify-no-pickle-6166-4: Deprecate pickling support in twisted.internet._sslverfiy.

Author: canibanoglu, tom.prince
Reviewer: exarkun, tom.prince, rwall
Fixes: #6166

Pickling twisted.internet.ssl.OptionSSLCertificationOptions and
twisted.internet.ssl.Keypair is no longer supported.

Note: See TracTickets for help on using tickets.