[Twisted-Python] Thread Cancelled

[Colin Dunklau]

On Sun, Jan 24, 2021 at 11:45 AM Robert DiFalco <robert.difalco at gmail.com>
wrote:

> Hi, I apologize this question is a little vague. I'm looking for pointers.
> I have a klein route that makes an underlying deferToThread call with a
> simple single thread (an IO based sync call I can't change, a boto3 sqs
> write). The thread pool is simple, just a couple of threads, nothing fancy.
>
> VERY rarely it appears that Klein cancels the thread. What techniques can
> I use to figure out why my thread is being Canceled? There's nothing in the
> failure to tell me "who, why, or where" it was canceled. Also, I cannot get
> this down to a reproducible case, but here's the boto3 sqs wrapper, this
> fall back works fine, but it's a band-aide for an error I can't track down.:
>
> def write(self, payload):
>     """
>     Write message to SQS async from thread pool. If twisted cancels the
>     thread, instead write synchronously.
>
>     def _retrySynchronously(error):
>         if error.type != CancelledError:
>             return error
>
>         log.warn("Async SQS write cancelled. Calling synchronously.")
>         return defer.succeed(self._writeSyncFallback(payload))
>
>     deferredCall = self._deferToThread(self.sqs.write, payload)
>     deferredCall.addErrback(_retrySynchronously)
>     return deferredCall
>
> def _writeSyncFallback(self, payload):
>     return self.sqs.write(payload)
>
> The _deferToThread call just uses my own thread pool with 2 threads, but
> is otherwise stock.
>
> Is there a level of logging I'm missing or some other thing that would
> tell me why the thread is being canceled? The retry works great and Klein
> does not return an error from the route.
>
> Thanks in advance.
>
>
I think we'll need to see more code for this, specifically the caller of
that `write` method, and its callers, etc. Note that the thread itself
isn't being cancelled, the Deferred you get from _deferToThread is... so
you'll most likely need to find out what code interacts with that object to
progress in isolating this.

In my quick skim of the deferToThread and ThreadPool source, I can't find
any explicit cancellations. While that certainly doesn't rule it out, it
does make me think you're more likely to find the issue by inspecting the
callers involved.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20210124/55094bb6/attachment.htm>