[Twisted-Python] module interface checking tool

James Y Knight foom at fuhm.net
Thu Aug 5 03:06:49 EDT 2004


So, I was a bit irritated recently because there is no way to test for 
private name usage in a module's namespace. '__all__' is a good start, 
but it is somewhat poor for a few reasons:
1) it doesn't affect import modname; modname.someprivatename at all so 
you may be using private interfaces without having any way of knowing.
2) Because it doesn't actually affect code people write, it often 
doesn't get updated when new public interfaces get added.
3) until recently it didn't even affect help(module), so users don't 
even know they're using a private name.

So, I wrote a tool to do the access checking. My goal is to have it 
used for testing (solving problems 1&2 *within* twisted, at least), not 
as something that you'd run your production server under.

sandbox/foom/private.py. Currently, it just spits out warnings because 
that way I can actually run through all the twisted tests, but that's 
controllable as a flag. Ideally every module in twisted would define 
__all__ and we could run the entire testsuite with access control 
enabled without raising illegal access exceptions.

To run the twisted tests with this, I temporarily stuck the following 
in twisted/__init__.py:
import private
private.privatizeModulesWithPrefix('twisted')

Much of the output (attached at the end) is from the IMAP tests trying 
to access private bits in order to test them. I'm not sure exactly what 
to do about that -- tests _should_ sometimes be able to access private 
parts, but they should also be restricted, normally, because you want 
to test against the public interface as well. So I may have to create a 
way to annotate some tests as being able to access private attributes 
while disallowing it by default.

Also, rebuild accesses private classes, and it is supposed to, so 
should be given special dispensation.

Besides those, the other things that show up currently are:
twisted.internet.abstract.isIPAddress from 
twisted.internet.iocpreactor.tcp (?)
twisted.internet.protocol.ConsumerToProtocolAdapter from 
twisted.protocols.ftp (?)
twisted.internet.threads._putResultInDeferred from 
twisted.enterprise.adbapi (_deferToThread)

Some of those perhaps should be public (?), but _putResultInDeferred, 
probably should not. IMO these are bugs, either of omission in __all__ 
or access from an inappropriate place. I dunno which.

Also, only 35 out of 559 modules in twisted use __all__. That's sad, 
and should be fixed. I'm sure if that was done, more of these kinds of 
things would show up.

Here's the full list of accesses to private names (I took the output 
and sort|uniq'd it). '=> Yes' means access was granted based on the two 
modules living in the same package. '=> No' means it would have raised 
an exception if I had that enabled.

<= private twisted.internet.abstract.isIPAddress from 
twisted.internet.iocpreactor.tcp (?): => No
<= private twisted.internet.protocol.BaseProtocol from 
twisted.python.rebuild (latestClass): => No
<= private twisted.internet.protocol.ConsumerToProtocolAdapter from 
twisted.protocols.ftp (?): => No
<= private twisted.internet.threads._putResultInDeferred from 
twisted.enterprise.adbapi (_deferToThread): => No
<= private twisted.mail.imap4.Command from twisted.mail.test.test_imap 
(append): => No
<= private twisted.mail.imap4.DontQuoteMe from 
twisted.mail.test.test_imap (testQuoteAvoider): => No
<= private twisted.mail.imap4.FileProducer from 
twisted.mail.test.test_imap (testFileProducer): => No
<= private twisted.mail.imap4.MessageProducer from 
twisted.mail.test.test_imap (testMultipleMultiPart): => No
<= private twisted.mail.imap4.MessageProducer from 
twisted.mail.test.test_imap (testSingleMultiPart): => No
<= private twisted.mail.imap4.MessageProducer from 
twisted.mail.test.test_imap (testSinglePart): => No
<= private twisted.mail.imap4.MessageSet from 
twisted.mail.test.test_imap (?): => No
<= private twisted.mail.imap4.MessageSet from 
twisted.mail.test.test_imap (testSetFlags): => No
<= private twisted.mail.imap4._FetchParser from 
twisted.mail.test.test_imap (testFetchParserBody): => No
<= private twisted.mail.imap4._FetchParser from 
twisted.mail.test.test_imap (testFetchParserMacros): => No
<= private twisted.mail.imap4._FetchParser from 
twisted.mail.test.test_imap (testFetchParserSimple): => No
<= private twisted.mail.imap4._formatHeaders from 
twisted.mail.test.test_imap (testFetchHeaders): => No
<= private twisted.mail.imap4._formatHeaders from 
twisted.mail.test.test_imap (testHeaderFormatter): => No
<= private twisted.mail.imap4.collapseNestedLists from 
twisted.mail.test.test_imap (testFiles): => No
<= private twisted.mail.imap4.collapseNestedLists from 
twisted.mail.test.test_imap (testQuoteAvoider): => No
<= private twisted.mail.imap4.collapseStrings from 
twisted.mail.test.test_imap (testStringCollapser): => No
<= private twisted.mail.imap4.decoder from twisted.mail.test.test_imap 
(testDecode): => No
<= private twisted.mail.imap4.parseIdList from 
twisted.mail.test.test_imap (testIdListParser): => No
<= private twisted.mail.imap4.parseNestedParens from 
twisted.mail.test.test_imap (_searchWork): => No
<= private twisted.mail.imap4.parseNestedParens from 
twisted.mail.test.test_imap (testLiterals): => No
<= private twisted.mail.imap4.parseNestedParens from 
twisted.mail.test.test_imap (testParenParser): => No
<= private twisted.mail.imap4.splitQuoted from 
twisted.mail.test.test_imap (testQuotedSplitter): => No
<= private twisted.mail.imap4.wildcardToRegexp from 
twisted.mail.test.test_imap (testWildcard): => No
<= private twisted.mail.imap4.wildcardToRegexp from 
twisted.mail.test.test_imap (testWildcardNoDelim): => No
<= private twisted.persisted.dirdbm.ALLOW_TWISTED_REBUILD from 
twisted.python.rebuild (rebuild): => No
<= private twisted.web.xmlrpc.addIntrospection from 
twisted.web.test.test_xmlrpc (?): => No

<= private twisted.internet.abstract.isIPAddress from 
twisted.internet.base (resolve): => Yes
<= private twisted.internet.abstract.isIPAddress from 
twisted.internet.tcp (resolveAddress): => Yes
<= private twisted.internet.abstract.isIPAddress from 
twisted.internet.udp (connect): => Yes
<= private twisted.internet.abstract.isIPAddress from 
twisted.internet.udp (startListening): => Yes
<= private twisted.internet.base.BaseConnector from 
twisted.internet.ssl (?): => Yes
<= private twisted.internet.base.BaseConnector from 
twisted.internet.ssl (__init__): => Yes
<= private twisted.internet.base.BaseConnector from 
twisted.internet.tcp (?): => Yes

James





More information about the Twisted-Python mailing list