[Twisted-Python] module interface checking tool
James Y Knight
foom at fuhm.net
Thu Aug 5 03:06:49 EDT 2004
So, I was a bit irritated recently because there is no way to test for
private name usage in a module's namespace. '__all__' is a good start,
but it is somewhat poor for a few reasons:
1) it doesn't affect import modname; modname.someprivatename at all so
you may be using private interfaces without having any way of knowing.
2) Because it doesn't actually affect code people write, it often
doesn't get updated when new public interfaces get added.
3) until recently it didn't even affect help(module), so users don't
even know they're using a private name.
So, I wrote a tool to do the access checking. My goal is to have it
used for testing (solving problems 1&2 *within* twisted, at least), not
as something that you'd run your production server under.
sandbox/foom/private.py. Currently, it just spits out warnings because
that way I can actually run through all the twisted tests, but that's
controllable as a flag. Ideally every module in twisted would define
__all__ and we could run the entire testsuite with access control
enabled without raising illegal access exceptions.
To run the twisted tests with this, I temporarily stuck the following
in twisted/__init__.py:
import private
private.privatizeModulesWithPrefix('twisted')
Much of the output (attached at the end) is from the IMAP tests trying
to access private bits in order to test them. I'm not sure exactly what
to do about that -- tests _should_ sometimes be able to access private
parts, but they should also be restricted, normally, because you want
to test against the public interface as well. So I may have to create a
way to annotate some tests as being able to access private attributes
while disallowing it by default.
Also, rebuild accesses private classes, and it is supposed to, so
should be given special dispensation.
Besides those, the other things that show up currently are:
twisted.internet.abstract.isIPAddress from
twisted.internet.iocpreactor.tcp (?)
twisted.internet.protocol.ConsumerToProtocolAdapter from
twisted.protocols.ftp (?)
twisted.internet.threads._putResultInDeferred from
twisted.enterprise.adbapi (_deferToThread)
Some of those perhaps should be public (?), but _putResultInDeferred,
probably should not. IMO these are bugs, either of omission in __all__
or access from an inappropriate place. I dunno which.
Also, only 35 out of 559 modules in twisted use __all__. That's sad,
and should be fixed. I'm sure if that was done, more of these kinds of
things would show up.
Here's the full list of accesses to private names (I took the output
and sort|uniq'd it). '=> Yes' means access was granted based on the two
modules living in the same package. '=> No' means it would have raised
an exception if I had that enabled.
<= private twisted.internet.abstract.isIPAddress from
twisted.internet.iocpreactor.tcp (?): => No
<= private twisted.internet.protocol.BaseProtocol from
twisted.python.rebuild (latestClass): => No
<= private twisted.internet.protocol.ConsumerToProtocolAdapter from
twisted.protocols.ftp (?): => No
<= private twisted.internet.threads._putResultInDeferred from
twisted.enterprise.adbapi (_deferToThread): => No
<= private twisted.mail.imap4.Command from twisted.mail.test.test_imap
(append): => No
<= private twisted.mail.imap4.DontQuoteMe from
twisted.mail.test.test_imap (testQuoteAvoider): => No
<= private twisted.mail.imap4.FileProducer from
twisted.mail.test.test_imap (testFileProducer): => No
<= private twisted.mail.imap4.MessageProducer from
twisted.mail.test.test_imap (testMultipleMultiPart): => No
<= private twisted.mail.imap4.MessageProducer from
twisted.mail.test.test_imap (testSingleMultiPart): => No
<= private twisted.mail.imap4.MessageProducer from
twisted.mail.test.test_imap (testSinglePart): => No
<= private twisted.mail.imap4.MessageSet from
twisted.mail.test.test_imap (?): => No
<= private twisted.mail.imap4.MessageSet from
twisted.mail.test.test_imap (testSetFlags): => No
<= private twisted.mail.imap4._FetchParser from
twisted.mail.test.test_imap (testFetchParserBody): => No
<= private twisted.mail.imap4._FetchParser from
twisted.mail.test.test_imap (testFetchParserMacros): => No
<= private twisted.mail.imap4._FetchParser from
twisted.mail.test.test_imap (testFetchParserSimple): => No
<= private twisted.mail.imap4._formatHeaders from
twisted.mail.test.test_imap (testFetchHeaders): => No
<= private twisted.mail.imap4._formatHeaders from
twisted.mail.test.test_imap (testHeaderFormatter): => No
<= private twisted.mail.imap4.collapseNestedLists from
twisted.mail.test.test_imap (testFiles): => No
<= private twisted.mail.imap4.collapseNestedLists from
twisted.mail.test.test_imap (testQuoteAvoider): => No
<= private twisted.mail.imap4.collapseStrings from
twisted.mail.test.test_imap (testStringCollapser): => No
<= private twisted.mail.imap4.decoder from twisted.mail.test.test_imap
(testDecode): => No
<= private twisted.mail.imap4.parseIdList from
twisted.mail.test.test_imap (testIdListParser): => No
<= private twisted.mail.imap4.parseNestedParens from
twisted.mail.test.test_imap (_searchWork): => No
<= private twisted.mail.imap4.parseNestedParens from
twisted.mail.test.test_imap (testLiterals): => No
<= private twisted.mail.imap4.parseNestedParens from
twisted.mail.test.test_imap (testParenParser): => No
<= private twisted.mail.imap4.splitQuoted from
twisted.mail.test.test_imap (testQuotedSplitter): => No
<= private twisted.mail.imap4.wildcardToRegexp from
twisted.mail.test.test_imap (testWildcard): => No
<= private twisted.mail.imap4.wildcardToRegexp from
twisted.mail.test.test_imap (testWildcardNoDelim): => No
<= private twisted.persisted.dirdbm.ALLOW_TWISTED_REBUILD from
twisted.python.rebuild (rebuild): => No
<= private twisted.web.xmlrpc.addIntrospection from
twisted.web.test.test_xmlrpc (?): => No
<= private twisted.internet.abstract.isIPAddress from
twisted.internet.base (resolve): => Yes
<= private twisted.internet.abstract.isIPAddress from
twisted.internet.tcp (resolveAddress): => Yes
<= private twisted.internet.abstract.isIPAddress from
twisted.internet.udp (connect): => Yes
<= private twisted.internet.abstract.isIPAddress from
twisted.internet.udp (startListening): => Yes
<= private twisted.internet.base.BaseConnector from
twisted.internet.ssl (?): => Yes
<= private twisted.internet.base.BaseConnector from
twisted.internet.ssl (__init__): => Yes
<= private twisted.internet.base.BaseConnector from
twisted.internet.tcp (?): => Yes
James
More information about the Twisted-Python
mailing list