Intermittent fcntl failure on FreeBSD

[exarkun]

A couple tests failed on the FreeBSD builder recently:

===============================================================================
[ERROR]: twisted.internet.test.test_process.PTYProcessTestsBuilder_PollReactor.test_spawnProcessTooEarlyWarns

Traceback (most recent call last):
  File "/usr/home/wiqd/twisted/freebsd-py2.3/Twisted/twisted/internet/test/test_process.py", line 48, in test_spawnProcessTooEarlyWarns
    self.spawnProcess, reactor)
  File "/usr/home/wiqd/twisted/freebsd-py2.3/Twisted/twisted/trial/unittest.py", line 373, in failUnlessWarns
    result = f(*args, **kwargs)
  File "/usr/home/wiqd/twisted/freebsd-py2.3/Twisted/twisted/internet/test/test_process.py", line 32, in spawnProcess
    usePTY=self.usePTY)
  File "/usr/home/wiqd/twisted/freebsd-py2.3/Twisted/twisted/internet/posixbase.py", line 221, in spawnProcess
    processProtocol, uid, gid, usePTY)
  File "/usr/home/wiqd/twisted/freebsd-py2.3/Twisted/twisted/internet/process.py", line 826, in __init__
    fdesc.setNonBlocking(masterfd)
  File "/usr/home/wiqd/twisted/freebsd-py2.3/Twisted/twisted/internet/fdesc.py", line 30, in setNonBlocking
    fcntl.fcntl(fd, FCNTL.F_SETFL, flags)
exceptions.IOError: [Errno 35] Resource temporarily unavailable
===============================================================================
[ERROR]: twisted.internet.test.test_process.PTYProcessTestsBuilder_SelectReactor.test_callWhenRunningSpawnProcessWarningFree

Traceback (most recent call last):
  File "/usr/home/wiqd/twisted/freebsd-py2.3/Twisted/twisted/internet/base.py", line 387, in _continueFiring
    callable(*args, **kwargs)
  File "/usr/home/wiqd/twisted/freebsd-py2.3/Twisted/twisted/internet/test/test_process.py", line 32, in spawnProcess
    usePTY=self.usePTY)
  File "/usr/home/wiqd/twisted/freebsd-py2.3/Twisted/twisted/internet/posixbase.py", line 221, in spawnProcess
    processProtocol, uid, gid, usePTY)
  File "/usr/home/wiqd/twisted/freebsd-py2.3/Twisted/twisted/internet/process.py", line 826, in __init__
    fdesc.setNonBlocking(masterfd)
  File "/usr/home/wiqd/twisted/freebsd-py2.3/Twisted/twisted/internet/fdesc.py", line 30, in setNonBlocking
    fcntl.fcntl(fd, FCNTL.F_SETFL, flags)
exceptions.IOError: [Errno 35] Resource temporarily unavailable
-------------------------------------------------------------------------------

This failure mode of fcntl is mentioned on #336 and #338, but there's enough other noise on those tickets that I'm not going to try to re-open either of them.

It seems most likely that we never fixed this (the problem seemed to go away by itself, ha ha) and there is just a timing component to this failure which is once again playing out so as to produce the problem.

I still have no idea why fcntl fails with EAGAIN on FreeBSD. Perhaps someone can dive into some FreeBSD documentation or source code and find out.

[glyph]

A brief dive into the documentation suggests that FreeBSD is just buggy. Quoting from  the errors section of the `fcntl` manpage helpfully published by freebsd.org:

     [EAGAIN]		The argument cmd is F_SETLK, the type of lock (l_type)
			is a shared lock (F_RDLCK) or exclusive lock
			(F_WRLCK), and the segment of a file to be locked is
			already exclusive-locked by another process; or the
			type is an exclusive lock and some portion of the seg-
			ment of a file to be locked is already shared-locked
			or exclusive-locked by another process.

Perhaps worthwhile to get in touch with some FreeBSD kernel developers?