[Twisted-Python] Thread Cancelled

[Robert DiFalco]

You're absolutely right, I meant "cancel the deferred". I don't grok server
sockets very well so maybe someone can help. But apparently, klein does a
.doRead from our server socket (getting the request from the client?). This
returns a "why" of "connection done" so that closes the connection before
we have written our response to the client, and that cancels the deferred
SQS write.

https://github.com/racker/python-twisted-core/blob/master/twisted/internet/selectreactor.py#L148-L155

The method above is "doRead". Which calls this:
https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L239

I guess if If socket.rcv() returns an empty string it simply closes the
connection.
https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L249-L250

Is that normal? I mean I guess it must be but then why is the read getting
an empty string and closing the connection? I can't really account for it?
Some kind of back pressure due to load?

Thanks for any thoughts.



On Sun, Jan 24, 2021 at 11:32 AM Colin Dunklau <colin.dunklau at gmail.com>
wrote:

>
>
> On Sun, Jan 24, 2021 at 11:45 AM Robert DiFalco <robert.difalco at gmail.com>
> wrote:
>
>> Hi, I apologize this question is a little vague. I'm looking for
>> pointers. I have a klein route that makes an underlying deferToThread call
>> with a simple single thread (an IO based sync call I can't change, a boto3
>> sqs write). The thread pool is simple, just a couple of threads, nothing
>> fancy.
>>
>> VERY rarely it appears that Klein cancels the thread. What techniques can
>> I use to figure out why my thread is being Canceled? There's nothing in the
>> failure to tell me "who, why, or where" it was canceled. Also, I cannot get
>> this down to a reproducible case, but here's the boto3 sqs wrapper, this
>> fall back works fine, but it's a band-aide for an error I can't track down.:
>>
>> def write(self, payload):
>>     """
>>     Write message to SQS async from thread pool. If twisted cancels the
>>     thread, instead write synchronously.
>>
>>     def _retrySynchronously(error):
>>         if error.type != CancelledError:
>>             return error
>>
>>         log.warn("Async SQS write cancelled. Calling synchronously.")
>>         return defer.succeed(self._writeSyncFallback(payload))
>>
>>     deferredCall = self._deferToThread(self.sqs.write, payload)
>>     deferredCall.addErrback(_retrySynchronously)
>>     return deferredCall
>>
>> def _writeSyncFallback(self, payload):
>>     return self.sqs.write(payload)
>>
>> The _deferToThread call just uses my own thread pool with 2 threads, but
>> is otherwise stock.
>>
>> Is there a level of logging I'm missing or some other thing that would
>> tell me why the thread is being canceled? The retry works great and Klein
>> does not return an error from the route.
>>
>> Thanks in advance.
>>
>>
> I think we'll need to see more code for this, specifically the caller of
> that `write` method, and its callers, etc. Note that the thread itself
> isn't being cancelled, the Deferred you get from _deferToThread is... so
> you'll most likely need to find out what code interacts with that object to
> progress in isolating this.
>
> In my quick skim of the deferToThread and ThreadPool source, I can't find
> any explicit cancellations. While that certainly doesn't rule it out, it
> does make me think you're more likely to find the issue by inspecting the
> callers involved.
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20210124/23d4c670/attachment.htm>