[Twisted-Python] finding out what's blocking the reactor

[Scott, Barry]

On Wednesday, 15 May 2019 06:27:30 BST Chris Withers wrote:
> On 14/05/2019 16:39, Scott, Barry wrote:
> > On Monday, 13 May 2019 15:26:01 BST Chris Withers wrote:
> >> Hi All,
> >> 
> >> On some busy instances of the app I'm working with/on, we see websocket
> >> connections being dropped due to not ping/ponging in time, I'm fairly
> >> sure this is as a result of some user-written callbacks blocking the
> >> reactor for long periods of time (seconds...).
> >> 
> >> What's the best way to debug a twisted application to find out what's
> >> blocking the reactor?
> > 
> > Is the process CPU bound or IO bound?
> 
> Yes ;-)
> 
> > If CPU bound then cprofile or the tool Glyph mentioned would help.
> > 
> > This advice is from a Linux perspective.
> > 
> > If its IO bound the you may be waiting on a blocking operation.
> > You could use strace on the process to see what its doing.
> 
> Good call on strace, will do so next time one of these happens.
> 
> > As its hung for seconds you could connect gdb to the process and see a
> > python backtrace using the py-bt command. You are likely to catch the
> > offending code.
> I'm failing miserably to get gdb to find debug symbols for python. I've
> installed the system python's debug symbols rpm, I've installed gdb .py
> that ships in the python source tree (so at least I *have* a py-bt
> command now!) - what am I likely to be missing?

At least on Fedora/Centos when you run up gdb you get a message that tells
you the the command to run to install missing symbols. This is what I see on
Fedora 30 for example:

$ gdb python3
GNU gdb (GDB) Fedora 8.2.91.20190424-24.fc30
... skip ...
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python3...
(No debugging symbols found in python3)
Missing separate debuginfos, use: dnf debuginfo-install 
python3-3.7.3-1.fc30.x86_64
(gdb) 

Then I quit gdb and run the sudo dnf as suggested.

Then next time you run gdb it will suggest even mode debug info to install.
You can often skip that, you have all you need to see python code.

> > You could adding logging to your code to show the time of callback entry
> > and exit.
> 
> Right, but if the callback yields, that will be falsely shown as
> blocking, no?

I'm not used to using yield with twisted. But it will not block.

> 
> cheers,
> 
> Chris