[Twisted-Python] TaskStopped error, unsure of the cause and solution
Paul Wiseman
poalman at gmail.com
Fri Feb 8 10:12:58 EST 2013
On 27 December 2012 14:49, Paul Wiseman <poalman at gmail.com> wrote:
> On 21 December 2012 18:08, <exarkun at twistedmatrix.com> wrote:
>> On 04:12 pm, poalman at gmail.com wrote:
>>>Hey,
>>>
>>>I'm getting the following error in my logs (lots and lots), but only
>>>after the server has been running for some time, I'm not sure what the
>>>cause is or what I can do to remedy it.
>>
>> This seems like a bug in `Agent`, perhaps related to `stopProducing`
>> being called more than once. It would be great if you could produce a
>> minimal example to include with a bug report in the issue tracker. An
>> example that reproduces the problem will also make it easier to see
>> where the bug is in your code, if it happens to be there rather than in
>> Twisted.
>
> I'll try my best! The problem is I don't seem to see this at all for
> 10k+ requests ish then I guess something must trigger it as it will
> start happening a lot, so it may be difficult to reproduce.
>
I've still not been able to reproduce this on demand.. the least
amount of request I've seen before getting the error is 6500 :/
I've also seen this stack trace which I didn't notice before
2013-02-08 10:07:28+0000 [HTTP11ClientProtocol,client] Unhandled Error
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/application/app.py",
line 323, in runReactorWithLogging
reactor.run()
File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/base.py",
line 1169, in run
self.mainLoop()
File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/base.py",
line 1181, in mainLoop
self.doIteration(t)
File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/epollreactor.py",
line 379, in doPoll
log.callWithLogger(selectable, _drdw, selectable, fd, event)
--- <exception caught here> ---
File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/python/log.py",
line 84, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/python/log.py",
line 69, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/python/context.py",
line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/python/context.py",
line 81, in callWithContext
return func(*args,**kw)
File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/posixbase.py",
line 639, in _doReadOrWrite
self._disconnectSelectable(selectable, why, inRead)
File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/posixbase.py",
line 258, in _disconnectSelectable
selectable.readConnectionLost(f)
File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/tcp.py",
line 267, in readConnectionLost
self.connectionLost(reason)
File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/tcp.py",
line 473, in connectionLost
self._commonConnection.connectionLost(self, reason)
File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/tcp.py",
line 281, in connectionLost
abstract.FileDescriptor.connectionLost(self, reason)
File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/abstract.py",
line 182, in connectionLost self.producer.stopProducing()
File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/web/client.py",
line 760, in stopProducing self._task.stop()
File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/task.py",
line 460, in stop self._completeWith(TaskStopped(),
Failure(TaskStopped()))
File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/task.py",
line 439, in _completeWith
self._cooperator._removeTask(self)
File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/task.py",
line 578, in _removeTask self._delayedCall.cancel()
File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/base.py",
line 89, in cancel
raise error.AlreadyCalled
twisted.internet.error.AlreadyCalled: Tried to cancel an
already-called event.
I'll keep trying to work out the issue, any suggestions on what I
might try to reproduce it or guesses at causes?
>>
>> Jean-Paul
>>>This is what I see in the logs:
>>>
>>>2012-12-20 17:07:17+0000 [HTTP11ClientProtocol,client] Unhandled Error
>>> Traceback (most recent call last):
>>> Failure: twisted.web._newclient.RequestTransmissionFailed:
>>>[<twisted.python.failure.Failure <class
>>>'twisted.internet.error.ConnectionDone'>>]
>>>
>>>2012-12-20 17:07:17+0000 [HTTP11ClientProtocol,client] Unexpected
>>>exception from twisted.web.client.FileBodyProducer.stopProducing
>>> Traceback (most recent call last):
>>> File "/usr/local/lib/python2.7/site-
>>>packages/Twisted-12.2.0-py2.7-linux-
>>>x86_64.egg/twisted/internet/endpoints.py",
>>>line 99, in connectionLost
>>> return self._wrappedProtocol.connectionLost(reason)
>>> File "/usr/local/lib/python2.7/site-
>>>packages/Twisted-12.2.0-py2.7-linux-
>>>x86_64.egg/twisted/web/_newclient.py",
>>>line 859, in dispatcher
>>> return func(*args, **kwargs)
>>> File "/usr/local/lib/python2.7/site-
>>>packages/Twisted-12.2.0-py2.7-linux-
>>>x86_64.egg/twisted/web/_newclient.py",
>>>line 1458, in _connectionLost_TRANSMITTING
>>> self._currentRequest.stopWriting()
>>> File "/usr/local/lib/python2.7/site-
>>>packages/Twisted-12.2.0-py2.7-linux-
>>>x86_64.egg/twisted/web/_newclient.py",
>>>line 760, in stopWriting
>>> _callAppFunction(self.bodyProducer.stopProducing)
>>> --- <exception caught here> ---
>>> File "/usr/local/lib/python2.7/site-
>>>packages/Twisted-12.2.0-py2.7-linux-
>>>x86_64.egg/twisted/web/_newclient.py",
>>>line 191, in _callAppFunction
>>> function()
>>> File "/usr/local/lib/python2.7/site-
>>>packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/web/client.py",
>>>line 760, in stopProducing
>>> self._task.stop()
>>> File "/usr/local/lib/python2.7/site-
>>>packages/Twisted-12.2.0-py2.7-linux-
>>>x86_64.egg/twisted/internet/task.py",
>>>line 459, in stop
>>> self._checkFinish()
>>> File "/usr/local/lib/python2.7/site-
>>>packages/Twisted-12.2.0-py2.7-linux-
>>>x86_64.egg/twisted/internet/task.py",
>>>line 469, in _checkFinish
>>> raise self._completionState
>>> twisted.internet.task.TaskStopped:
>>>
>>>Just loads of those, they always seem to come through in pairs (the
>>>first twisted.internet.error.ConnectionDone, and then the
>>>twisted.internet.task.TaskStopped as above)
>>>
>>>I'm just doing PUT requests with twisted.web.client.Agent and using a
>>>twisted.web.client.FileBodyProducer to provide the content.
>>>
>>>I'm doing the PUT while I still have the request from a render_GET, so
>>>I thought maybe it's caused if the user disconnects while I'm doing
>>>the PUT? The idea is I redirect them to the file I PUT with
>>>request.redirect, which seems to work fine (until I get this error and
>>>it doesn't)
>>>
>>>The reason I do a PUT and redirect, instead of just writing back to
>>>the request is that I check if the file is in a cache first, I then
>>>only PUT if it's not there- then redirect afterwards.
>>>
>>>Any clues or ideas of what is happening would be hugely helpful
>>>
>>>thanks!!
>>>
>>>Paul
>>>
>>>_______________________________________________
>>>Twisted-Python mailing list
>>>Twisted-Python at twistedmatrix.com
>>>http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
>> _______________________________________________
>> Twisted-Python mailing list
>> Twisted-Python at twistedmatrix.com
>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
More information about the Twisted-Python
mailing list