[Twisted-Python] finding out what's blocking the reactor

[Glyph]

> On May 14, 2019, at 8:39 AM, Scott, Barry <barry.scott at forcepoint.com> wrote:
> 
> On Monday, 13 May 2019 15:26:01 BST Chris Withers wrote:
>> Hi All,
>> 
>> On some busy instances of the app I'm working with/on, we see websocket
>> connections being dropped due to not ping/ponging in time, I'm fairly
>> sure this is as a result of some user-written callbacks blocking the
>> reactor for long periods of time (seconds...).
>> 
>> What's the best way to debug a twisted application to find out what's
>> blocking the reactor?
> 
> Is the process CPU bound or IO bound?
> 
> If CPU bound then cprofile or the tool Glyph mentioned would help.

The tool I mentioned (vmprof) is not a CPU profiler, and will help with I/O bound tasks as well.  cProfile can also tell you about wall clock time, but has other limitations that might not make it suitable for this.

-g

> This advice is from a Linux perspective.
> 
> If its IO bound the you may be waiting on a blocking operation.
> You could use strace on the process to see what its doing.
> 
> As its hung for seconds you could connect gdb to the process and see a python 
> backtrace using the py-bt command. You are likely to catch the offending code.
> 
> The normal bt command will show you what the last python command it calling.
> 
> You could adding logging to your code to show the time of callback entry and 
> exit.
> 
> If all else fails patch the twisted code to time callbacks and report the slow 
> ones.
> 
> Barry
> 
>> 
>> cheers,
>> 
>> Chris
>> 
>> _______________________________________________
>> Twisted-Python mailing list
>> Twisted-Python at twistedmatrix.com
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> 
> 
> 
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python