[Twisted-Python] Twisted and Thread, thread not running when twisted app is running as service

Jean-Paul Calderone exarkun at twistedmatrix.com
Fri May 22 07:39:54 MDT 2020


On Thu, May 21, 2020 at 4:46 PM Sereysethy TOUCH <touch.sereysethy at gmail.com>
wrote:

> I am on MacOS, it is a development phase, but deployment will be on Linux.
>
> It is hard to run dtrace/dtruss on MacOS due to the "system integrity
> protection" on MacOS. I got some output but very limited information.
>
> So if there is a workaround you might suggest?
>

I would definitely recommend nailing down the exact cause of the problem.
I think that system integrity protection can be bypassed, perhaps by
creating a virtualenv?  I am not particularly familiar with macOS but I'm
sure there's some good information somewhere out there.

If the problem is the blocking network operations performed by
`PolicyClient` then you'll probably have to move even the instantiation of
that class to a separate thread.  This is probably good anyway, since it's
not necessarily the case that it is safe to create the object in one thread
and then use it in another.  Better to create and use it all in the same
thread.

Jean-Paul


>
> TS
>
> On Thu, May 21, 2020 at 10:20 PM Jean-Paul Calderone <
> exarkun at twistedmatrix.com> wrote:
>
>> On Thu, May 21, 2020 at 3:56 PM Sereysethy TOUCH <
>> touch.sereysethy at gmail.com> wrote:
>>
>>> Hi Jean-Paul,
>>>
>>> I found the error after working on a short, self contained, correcte
>>> example. It is not about twisted and normal thread in general, but
>>> something else.
>>>
>>> After more tests, I come to realise that whenever I instantiate this
>>> class PolicyClient from this library
>>> https://github.com/ray-project/ray/blob/master/rllib/env/policy_client.py in
>>> my protocol or just anywhere, twisted app either freezes or terminates, and
>>> this *only* happens when the app runs as a daemon, created using a
>>> service (.tac file).
>>>
>>> I hope you can take a look at the class PolicyClient, why it causes
>>> problem to reactor main loop. Please point me to where the idea might come
>>> from.
>>>
>>> I can give you the example but not sure if it is enough, as the client
>>> needs to connect to a server.
>>>
>>
>> It looks like PolicyClient does a blocking HTTP request in __init__.
>> Since PolicyClient is instantiated in the reactor thread, this will block
>> the reactor.  Perhaps this call is hanging for some reason when the process
>> has daemonized?  This would be unusual but not unheard of.  For example,
>> there are macOS environments where a process does not have access to the
>> network if it is not associated with a windowing session.
>>
>> What platform do you observe the problem on, and what more can you learn
>> about what the process does on its way to hanging (eg can you run strace on
>> it)?
>>
>> Jean-Paul
>>
>>
>>>
>>> Thank you,
>>> TS
>>>
>>>
>>> On Thu, May 21, 2020 at 5:23 PM Jean-Paul Calderone <
>>> exarkun at twistedmatrix.com> wrote:
>>>
>>>> On Thu, May 21, 2020 at 10:58 AM Sereysethy TOUCH <
>>>> touch.sereysethy at gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I am developing a twisted app, and it runs as a service using twistd
>>>>> -y to start the app.
>>>>> I am having a problem of running a thread. I know it is not
>>>>> recommended to use thread, but the library that I use, the object created
>>>>> is running in a thread.
>>>>>
>>>>> Here is the problem:
>>>>>
>>>>> 1) if I start reactor by running reactor.run() directly, thread is
>>>>> running fine
>>>>> 2) if I run it as a service using twisted, the thread is not running,
>>>>> it runs but it seems to be blocked, because I tried to write something to
>>>>> file using time.sleep(), but file is empty.
>>>>>
>>>>> Is there something that I miss? How can I debug this?
>>>>>
>>>>
>>>> A good next step would be to create at Short, Self Contained, Correct
>>>> (Compilable), Example <http://sscce.org/> and share it.
>>>>
>>>> Jean-Paul
>>>>
>>>>
>>>>>
>>>>> Thank you,
>>>>>
>>>>> TS
>>>>> _______________________________________________
>>>>> Twisted-Python mailing list
>>>>> Twisted-Python at twistedmatrix.com
>>>>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>>>>
>>>> _______________________________________________
>>>> Twisted-Python mailing list
>>>> Twisted-Python at twistedmatrix.com
>>>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>>>
>>> _______________________________________________
>>> Twisted-Python mailing list
>>> Twisted-Python at twistedmatrix.com
>>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>>
>> _______________________________________________
>> Twisted-Python mailing list
>> Twisted-Python at twistedmatrix.com
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20200522/ee4ab853/attachment.htm>


More information about the Twisted-Python mailing list