[Twisted-Python] kqueue reactor / ticket #1918

Tobias Oberstein tobias.oberstein at tavendo.de
Sat Nov 5 14:46:03 EDT 2011


> > Then, baseline. I've run the trial using select reactor in the
> > expectation of finding all tests passed.
> >
> > This is not the case, there are a couple of failed ones. Those are
> > also failed with the kqueue reactor, so I need to know whether I need
> > to inspect those or those are problems unrelated to the reactor, but of
> general (platform?) nature.
> 
> Are you using Twisted 11? Many of these issues are, if I'm not mistaken,
> fixed in trunk (though perhaps not all). So trunk is probably a better baseline
> to work off of.

Ok, I'm using trunk now.

1)
The errors in

twisted.python.test.test_components

are gone.

2)
The errors in

twisted.internet.test.test_posixprocess
twisted.internet.test.test_process

are still there ... and they show up for select, poll and kqueue reactors.

They all somehow relate to "open FDs":

twisted.internet.test.test_posixprocess
  FileDescriptorTests
    test_expectedFDs ...                                                 [FAIL]

twisted.internet.test.test_process
  PTYProcessTestsBuilder_KQueueReactor
    test_openFileDescriptors ...                                        [ERROR]
  PTYProcessTestsBuilder_PollReactor
    test_openFileDescriptors ...                                        [ERROR]
  PTYProcessTestsBuilder_SelectReactor
    test_openFileDescriptors ...                                        [ERROR]
ProcessTestsBuilder_SelectReactor
    test_openFileDescriptors ...                                        [ERROR]

This might more a FreeBSD related thing. ?

3)
I think I've nailed down the Half-Close stuff.

4)
These are kqueue specific. Need to investigate further.

twisted.internet.test.test_fdset
  ReactorFDSetTestsBuilder_KQueueReactor
    test_lostFileDescriptor ...                                         [ERROR]
    test_removedFromReactor ...                                          [FAIL]

The first one comes up, since

removeReader()

is called for a FD which is gone,  and leads to a socket exception from fd = reader.fileno()

5)
These are kqueue specific. Need to investigate further.

twisted.conch.test.test_conch

The fail on the kqeue control call .. see below.

Any ideas what could be the reason?


[autobahn at autobahnhub ~/temp]$ trial -r select twisted.conch.test.test_conch
twisted.conch.test.test_conch
  CmdLineClientTestCase
    test_exec ...                                                          [OK]
    test_localToRemoteForwarding ...                                       [OK]
    test_remoteToLocalForwarding ...                                       [OK]
  OpenSSHClientForwardingTestCase
    test_exec ...                                                          [OK]
    test_localToRemoteForwarding ...                                       [OK]
    test_remoteToLocalForwarding ...                                       [OK]
  OpenSSHClientRekeyTestCase
    test_clientRekey ...                                                   [OK]
  StdioInteractingSessionTests
    test_eofReceived ...                                                   [OK]

-------------------------------------------------------------------------------
Ran 8 tests in 4.707s

PASSED (successes=8)
[autobahn at autobahnhub ~/temp]$ trial -r kqueue twisted.conch.test.test_conch
twisted.conch.test.test_conch
  CmdLineClientTestCase
    test_exec ...                                                          [OK]
    test_localToRemoteForwarding ...                                    [ERROR]
                                   [ERROR]
    test_remoteToLocalForwarding ...                                       [OK]
  OpenSSHClientForwardingTestCase
    test_exec ...                                                          [OK]
    test_localToRemoteForwarding ...                                    [ERROR]
                                   [ERROR]
    test_remoteToLocalForwarding ...                                       [OK]
  OpenSSHClientRekeyTestCase
    test_clientRekey ...                                                   [OK]
  StdioInteractingSessionTests
    test_eofReceived ...                                                   [OK]

===============================================================================
[ERROR]
Traceback (most recent call last):
Failure: twisted.internet.defer.TimeoutError: <twisted.conch.test.test_conch.CmdLineClientTestCase testMethod=test_localToRemoteForwarding> (test_localToRemoteForwarding) still running at 120.0 secs

twisted.conch.test.test_conch.CmdLineClientTestCase.test_localToRemoteForwarding
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/python/log.py", line 84, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/python/log.py", line 69, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/internet/kqreactor.py", line 225, in _doWriteOrRead
    self._disconnectSelectable(selectable, why, inRead)
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/internet/posixbase.py", line 260, in _disconnectSelectable
    selectable.readConnectionLost(f)
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/internet/tcp.py", line 257, in readConnectionLost
    self.connectionLost(reason)
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/internet/tcp.py", line 433, in connectionLost
    Connection.connectionLost(self, reason)
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/internet/tcp.py", line 277, in connectionLost
    protocol.connectionLost(reason)
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/conch/test/test_conch.py", line 242, in connectionLost
    self.protocol.forwardingPortDisconnected(self.buffer)
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/conch/test/test_conch.py", line 190, in forwardingPortDisconnected
    self.transport.write('\x03')
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/internet/process.py", line 862, in write
    self.pipes[0].write(data)
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/internet/process.py", line 174, in write
    abstract.FileDescriptor.write(self, data)
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/internet/abstract.py", line 310, in write
    self.startWriting()
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/internet/abstract.py", line 400, in startWriting
    self.reactor.addWriter(self)
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/internet/kqreactor.py", line 94, in addWriter
    self._updateRegistration(fd, KQ_FILTER_WRITE, KQ_EV_ADD)
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/internet/kqreactor.py", line 74, in _updateRegistration
    self._kq.control([kevent(*args)], 0, 0)
exceptions.OSError: [Errno 32] Broken pipe

twisted.conch.test.test_conch.CmdLineClientTestCase.test_localToRemoteForwarding
===============================================================================
[ERROR]
Traceback (most recent call last):
Failure: twisted.internet.defer.TimeoutError: <twisted.conch.test.test_conch.OpenSSHClientForwardingTestCase testMethod=test_localToRemoteForwarding> (test_localToRemoteForwarding) still running at 120.0 secs

twisted.conch.test.test_conch.OpenSSHClientForwardingTestCase.test_localToRemoteForwarding
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/python/log.py", line 84, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/python/log.py", line 69, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/internet/kqreactor.py", line 225, in _doWriteOrRead
    self._disconnectSelectable(selectable, why, inRead)
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/internet/posixbase.py", line 262, in _disconnectSelectable
    self.removeWriter(selectable)
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/internet/kqreactor.py", line 140, in removeWriter
    self._updateRegistration(fd, KQ_FILTER_WRITE, KQ_EV_DELETE)
  File "/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/internet/kqreactor.py", line 74, in _updateRegistration
    self._kq.control([kevent(*args)], 0, 0)
exceptions.OSError: [Errno 2] No such file or directory

twisted.conch.test.test_conch.OpenSSHClientForwardingTestCase.test_localToRemoteForwarding
-------------------------------------------------------------------------------
Ran 8 tests in 243.797s

FAILED (errors=4, successes=6)
[autobahn at autobahnhub ~/temp]$



More information about the Twisted-Python mailing list