Opened 8 years ago

Closed 7 months ago

#4644 defect closed fixed (fixed)

setuptools 0.6c11 (and maybe later?) causes test_openFileDescriptors to fail on OS X

Reported by: Glyph Owned by:
Priority: low Milestone:
Component: core Keywords:
Cc: Branch:


The exception is as follows:

Traceback (most recent call last):
  File "twisted/internet/", line 60, in maybeCallProcessEnded
  File "twisted/internet/test/", line 255, in processEnded
  File "twisted/internet/test/", line 242, in checkOutput
    self.assertEquals('[0, 1, 2, 3]', output)
twisted.trial.unittest.FailTest: not equal:
a = '[0, 1, 2, 3]'
b = '[0, 1, 2, 3, 6, 7, 8]'

This does not occur with the version of setuptools packaged in Snow Leopard, 0.6c9.

I haven't gotten all the way to the root cause, but I have investigated a little. zope.interface imports pkg_resources (to some nefarious end that I am sure I don't care at all about). This is updated by the more recent version of setuptools to do something more clever, I suppose. Observe:

$ python
Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.listdir('/dev/fd')
['0', '1', '2', '3']
>>> import pkg_resources
>>> os.listdir('/dev/fd')
['0', '1', '2', '3', '4', '5', '6']

I'm assuming that this has something to do with the fact that pkg_resources imports Carbon and MacOS and PyObjCTools and a bunch of other stuff. It was previously importing some of these same suspicious-looking modules but I guess it wasn't calling the API that opens these file descriptors.

I am assigning this to zooko because he seems to be our (perhaps unwitting and perhaps unwilling, but nevertheless somewhat effective) ambassador to the wild and wonderful world of distutils, setuptools, et. al..

However, I think we might want to fix the underlying problem here with the fragility of the test. A number of weird environment tricks could make it fail. For example, I'm pretty sure that the LD_PRELOAD shenanigans that authbind uses would cause a problem; if I recall correctly, it leaves open a pipe to talk to its setuid buddy. If not authbind, then something else which does something similar.

So, here's a first suggestion: we should be able to depend upon the fact that FDs 0, 1, and 2 are open. We should dup() those a couple of times in the subprocess, and verify that the duplicate values (A) show up now, and (B) didn't show up before the dup() calls. (While it's theoretically possible for some insane LD_PRELOAD or syscall tracing hook to silently allocate a bunch of FDs, then silently close them to replace them with something else as a result of only a dup() call, this strikes me as substantially less likely and less useful than simply having unexpected FDs other than ['0', '1', '2', '3'] open at startup or after importing some Twisted code.)

Change History (3)

comment:1 Changed 7 years ago by Glyph

The root cause of this issue is Python bug 7895.

comment:2 Changed 7 years ago by <automation>

Owner: zooko deleted

comment:3 Changed 7 months ago by Craig Rodrigues

Resolution: fixed
Status: newclosed

This doesn't seem to happen any more with contemporary versions of OS X and setuptools.

Note: See TracTickets for help on using tickets.