[Twisted-Python] finding out what's blocking the reactor

[Werner Thie]

Hi all

with periods of seemingly no activity the gc comes to mind.

Is the working set size stable or ever increasing, swap operations might
block activity of the process

I had problems in the beginning with twisted, because I was holding on
to a lot of objects and thus creating  circular non collectible object
references leading to ever increasing work sets over time, with the gc
hitting me hard once in a while.

First check I do always is keep an eye on the size of the process at
constant network load, size shrinking when network load is dwindling and
increasing when network load is growing. If it is possible to manipulate
network load precisely then returning to the same load should give you
the same process size after some time repeatedly.

As a reference I have twisted processes running with up times measured
in years on FreeBSD.

Werner

On 5/14/19 05:39, Scott, Barry wrote:
> On Monday, 13 May 2019 15:26:01 BST Chris Withers wrote:
>> Hi All,
>>
>> On some busy instances of the app I'm working with/on, we see websocket
>> connections being dropped due to not ping/ponging in time, I'm fairly
>> sure this is as a result of some user-written callbacks blocking the
>> reactor for long periods of time (seconds...).
>>
>> What's the best way to debug a twisted application to find out what's
>> blocking the reactor?
> Is the process CPU bound or IO bound?
>
> If CPU bound then cprofile or the tool Glyph mentioned would help.
>
> This advice is from a Linux perspective.
>
> If its IO bound the you may be waiting on a blocking operation.
> You could use strace on the process to see what its doing.
>
> As its hung for seconds you could connect gdb to the process and see a python 
> backtrace using the py-bt command. You are likely to catch the offending code.
>
> The normal bt command will show you what the last python command it calling.
>
> You could adding logging to your code to show the time of callback entry and 
> exit.
>
> If all else fails patch the twisted code to time callbacks and report the slow 
> ones.
>
> Barry
>
>> cheers,
>>
>> Chris
>>
>> _______________________________________________
>> Twisted-Python mailing list
>> Twisted-Python at twistedmatrix.com
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python