[Twisted-Python] TaskStopped error, unsure of the cause and solution

Paul Wiseman poalman at gmail.com
Fri Feb 8 10:12:58 EST 2013


On 27 December 2012 14:49, Paul Wiseman <poalman at gmail.com> wrote:
> On 21 December 2012 18:08,  <exarkun at twistedmatrix.com> wrote:
>> On 04:12 pm, poalman at gmail.com wrote:
>>>Hey,
>>>
>>>I'm getting the following error in my logs (lots and lots), but only
>>>after the server has been running for some time, I'm not sure what the
>>>cause is or what I can do to remedy it.
>>
>> This seems like a bug in `Agent`, perhaps related to `stopProducing`
>> being called more than once.  It would be great if you could produce a
>> minimal example to include with a bug report in the issue tracker.  An
>> example that reproduces the problem will also make it easier to see
>> where the bug is in your code, if it happens to be there rather than in
>> Twisted.
>
> I'll try my best! The problem is I don't seem to see this at all for
> 10k+ requests ish then I guess something must trigger it as it will
> start happening a lot, so it may be difficult to reproduce.
>

I've still not been able to reproduce this on demand.. the least
amount of request I've seen before getting the error is 6500 :/

I've also seen this stack trace which I didn't notice before

2013-02-08 10:07:28+0000 [HTTP11ClientProtocol,client] Unhandled Error
        Traceback (most recent call last):
          File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/application/app.py",
line 323, in runReactorWithLogging
            reactor.run()
          File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/base.py",
line 1169, in run
            self.mainLoop()
          File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/base.py",
line 1181, in mainLoop
            self.doIteration(t)
          File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/epollreactor.py",
line 379, in doPoll
            log.callWithLogger(selectable, _drdw, selectable, fd, event)
        --- <exception caught here> ---
          File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/python/log.py",
line 84, in callWithLogger
            return callWithContext({"system": lp}, func, *args, **kw)
          File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/python/log.py",
line 69, in callWithContext
            return context.call({ILogContext: newCtx}, func, *args, **kw)
          File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/python/context.py",
line 118, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/python/context.py",
line 81, in callWithContext
            return func(*args,**kw)
          File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/posixbase.py",
line 639, in _doReadOrWrite
            self._disconnectSelectable(selectable, why, inRead)
          File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/posixbase.py",
line 258, in _disconnectSelectable
            selectable.readConnectionLost(f)
          File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/tcp.py",
line 267, in readConnectionLost
            self.connectionLost(reason)
          File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/tcp.py",
line 473, in connectionLost
            self._commonConnection.connectionLost(self, reason)
          File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/tcp.py",
line 281, in connectionLost
abstract.FileDescriptor.connectionLost(self, reason)
          File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/abstract.py",
line 182, in connectionLost            self.producer.stopProducing()
          File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/web/client.py",
line 760, in stopProducing            self._task.stop()
          File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/task.py",
line 460, in stop            self._completeWith(TaskStopped(),
Failure(TaskStopped()))
          File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/task.py",
line 439, in _completeWith
self._cooperator._removeTask(self)
          File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/task.py",
line 578, in _removeTask            self._delayedCall.cancel()
          File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/base.py",
line 89, in cancel
            raise error.AlreadyCalled
        twisted.internet.error.AlreadyCalled: Tried to cancel an
already-called event.

I'll keep trying to work out the issue, any suggestions on what I
might try to reproduce it or guesses at causes?

>>
>> Jean-Paul
>>>This is what I see in the logs:
>>>
>>>2012-12-20 17:07:17+0000 [HTTP11ClientProtocol,client] Unhandled Error
>>>        Traceback (most recent call last):
>>>        Failure: twisted.web._newclient.RequestTransmissionFailed:
>>>[<twisted.python.failure.Failure <class
>>>'twisted.internet.error.ConnectionDone'>>]
>>>
>>>2012-12-20 17:07:17+0000 [HTTP11ClientProtocol,client] Unexpected
>>>exception from twisted.web.client.FileBodyProducer.stopProducing
>>>        Traceback (most recent call last):
>>>          File "/usr/local/lib/python2.7/site-
>>>packages/Twisted-12.2.0-py2.7-linux-
>>>x86_64.egg/twisted/internet/endpoints.py",
>>>line 99, in connectionLost
>>>            return self._wrappedProtocol.connectionLost(reason)
>>>          File "/usr/local/lib/python2.7/site-
>>>packages/Twisted-12.2.0-py2.7-linux-
>>>x86_64.egg/twisted/web/_newclient.py",
>>>line 859, in dispatcher
>>>            return func(*args, **kwargs)
>>>          File "/usr/local/lib/python2.7/site-
>>>packages/Twisted-12.2.0-py2.7-linux-
>>>x86_64.egg/twisted/web/_newclient.py",
>>>line 1458, in _connectionLost_TRANSMITTING
>>>            self._currentRequest.stopWriting()
>>>          File "/usr/local/lib/python2.7/site-
>>>packages/Twisted-12.2.0-py2.7-linux-
>>>x86_64.egg/twisted/web/_newclient.py",
>>>line 760, in stopWriting
>>>            _callAppFunction(self.bodyProducer.stopProducing)
>>>        --- <exception caught here> ---
>>>          File "/usr/local/lib/python2.7/site-
>>>packages/Twisted-12.2.0-py2.7-linux-
>>>x86_64.egg/twisted/web/_newclient.py",
>>>line 191, in _callAppFunction
>>>            function()
>>>          File "/usr/local/lib/python2.7/site-
>>>packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/web/client.py",
>>>line 760, in stopProducing
>>>            self._task.stop()
>>>          File "/usr/local/lib/python2.7/site-
>>>packages/Twisted-12.2.0-py2.7-linux-
>>>x86_64.egg/twisted/internet/task.py",
>>>line 459, in stop
>>>            self._checkFinish()
>>>          File "/usr/local/lib/python2.7/site-
>>>packages/Twisted-12.2.0-py2.7-linux-
>>>x86_64.egg/twisted/internet/task.py",
>>>line 469, in _checkFinish
>>>            raise self._completionState
>>>        twisted.internet.task.TaskStopped:
>>>
>>>Just loads of those, they always seem to come through in pairs (the
>>>first twisted.internet.error.ConnectionDone, and then the
>>>twisted.internet.task.TaskStopped as above)
>>>
>>>I'm just doing PUT requests with twisted.web.client.Agent and using a
>>>twisted.web.client.FileBodyProducer to provide the content.
>>>
>>>I'm doing the PUT while I still have the request from a render_GET, so
>>>I thought maybe it's caused if the user disconnects while I'm doing
>>>the PUT? The idea is I redirect them to the file I PUT with
>>>request.redirect, which seems to work fine (until I get this error and
>>>it doesn't)
>>>
>>>The reason I do a PUT and redirect, instead of just writing back to
>>>the request is that I check if the file is in a cache first, I then
>>>only PUT if it's not there- then redirect afterwards.
>>>
>>>Any clues or ideas of what is happening would be hugely helpful
>>>
>>>thanks!!
>>>
>>>Paul
>>>
>>>_______________________________________________
>>>Twisted-Python mailing list
>>>Twisted-Python at twistedmatrix.com
>>>http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
>> _______________________________________________
>> Twisted-Python mailing list
>> Twisted-Python at twistedmatrix.com
>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python



More information about the Twisted-Python mailing list