[Twisted-Python] Thread Cancelled

[Robert DiFalco]

That makes sense, thank you. A timeout seems unlikely but maybe the client
is closing the connection due to a network issue. This is an extremely rare
occurrence.

On Sun, Jan 24, 2021 at 2:41 PM Glyph <glyph at twistedmatrix.com> wrote:

> While a socket is open and receiving data, recv() will either give you a
> non-zero number of bytes if bytes are ready, or an EWOULDBLOCK (AKA EAGAIN)
> if no bytes are ready.  A result of zero bytes (the empty string) means
> "end of file" - the other end has closed the socket.
>
> So what's happening here is your client is timing out or otherwise
> canceling its request by closing the socket, and this is the correct,
> intentional response to that scenario.
>
> -g
>
> On Jan 24, 2021, at 11:57 AM, Robert DiFalco <robert.difalco at gmail.com>
> wrote:
>
> You're absolutely right, I meant "cancel the deferred". I don't grok
> server sockets very well so maybe someone can help. But apparently, klein
> does a .doRead from our server socket (getting the request from the
> client?). This returns a "why" of "connection done" so that closes the
> connection before we have written our response to the client, and that
> cancels the deferred SQS write.
>
>
> https://github.com/racker/python-twisted-core/blob/master/twisted/internet/selectreactor.py#L148-L155
>
> The method above is "doRead". Which calls this:
>
> https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L239
>
> I guess if If socket.rcv() returns an empty string it simply closes the
> connection.
>
> https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L249-L250
>
> Is that normal? I mean I guess it must be but then why is the read getting
> an empty string and closing the connection? I can't really account for it?
> Some kind of back pressure due to load?
>
> Thanks for any thoughts.
>
>
>
> On Sun, Jan 24, 2021 at 11:32 AM Colin Dunklau <colin.dunklau at gmail.com>
> wrote:
>
>>
>>
>> On Sun, Jan 24, 2021 at 11:45 AM Robert DiFalco <robert.difalco at gmail.com>
>> wrote:
>>
>>> Hi, I apologize this question is a little vague. I'm looking for
>>> pointers. I have a klein route that makes an underlying deferToThread call
>>> with a simple single thread (an IO based sync call I can't change, a boto3
>>> sqs write). The thread pool is simple, just a couple of threads, nothing
>>> fancy.
>>>
>>> VERY rarely it appears that Klein cancels the thread. What techniques
>>> can I use to figure out why my thread is being Canceled? There's nothing in
>>> the failure to tell me "who, why, or where" it was canceled. Also, I cannot
>>> get this down to a reproducible case, but here's the boto3 sqs wrapper,
>>> this fall back works fine, but it's a band-aide for an error I can't track
>>> down.:
>>>
>>> def write(self, payload):
>>>     """
>>>     Write message to SQS async from thread pool. If twisted cancels the
>>>     thread, instead write synchronously.
>>>
>>>     def _retrySynchronously(error):
>>>         if error.type != CancelledError:
>>>             return error
>>>
>>>         log.warn("Async SQS write cancelled. Calling synchronously.")
>>>         return defer.succeed(self._writeSyncFallback(payload))
>>>
>>>     deferredCall = self._deferToThread(self.sqs.write, payload)
>>>     deferredCall.addErrback(_retrySynchronously)
>>>     return deferredCall
>>>
>>> def _writeSyncFallback(self, payload):
>>>     return self.sqs.write(payload)
>>>
>>> The _deferToThread call just uses my own thread pool with 2 threads, but
>>> is otherwise stock.
>>>
>>> Is there a level of logging I'm missing or some other thing that would
>>> tell me why the thread is being canceled? The retry works great and Klein
>>> does not return an error from the route.
>>>
>>> Thanks in advance.
>>>
>>>
>> I think we'll need to see more code for this, specifically the caller of
>> that `write` method, and its callers, etc. Note that the thread itself
>> isn't being cancelled, the Deferred you get from _deferToThread is... so
>> you'll most likely need to find out what code interacts with that object to
>> progress in isolating this.
>>
>> In my quick skim of the deferToThread and ThreadPool source, I can't find
>> any explicit cancellations. While that certainly doesn't rule it out, it
>> does make me think you're more likely to find the issue by inspecting the
>> callers involved.
>> _______________________________________________
>> Twisted-Python mailing list
>> Twisted-Python at twistedmatrix.com
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20210124/5f2fa88c/attachment.htm>