[Twisted-Python] stop/start client connections with loseConnection in ReconnectingClientFactory

[Chris Satterthwaite]

Hi Chris,

The print and sleep statements are just my efforts at converting something big down to something small for a test case.  In the actual client I'm using twisted.logger for FilteringLogObserver's and Logger's.  The looping call for system health was also a shorted version; I wasn't sure if looping calls were the culprit at the start.  I appologise for the fluff and if I introduced confusion.  I just figured that posting multiple classes and thousands of lines would get less response than an attempted short test sample.


> when you say "When the client hits a problem, it calls transport.loseConnection()", where is that call to transport.loseConnection?

The transport.loseConnection call is on line 82 of the test client code; it's inside the cleanup() function.


> Now, as to what your problem is, I suspect it's this call to stopTrying:
> https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file-testclient-py-L87

I may be missing this from the Twisted side, but I understood that stopFactory was only called when the reactor was cleaning up.  So I added stopTrying in there for the ReconnectingClientFactory portion, amongst many other lines (that I removed for the example) for stopping external dependencies.  All that seemed to work fine.


When I called loseConnection with just a single established connection, stopFactory was called and ReconnectingClientFactory was not restarted or reconnected.  Seemed like ReconnectingClientFactory was not expecting a new connection attempt after the loseConnection call.  Additionally, it seemed like the factory expected to clean up and stop the reactor whenever loseConnection was called with that single connection.  And of course in most use-cases, that makes sense.  But my goal was to hang on to the factory after loseConnection, and continue work after external dependencies came back online.


> when you noticed some Twisted source code that works off factory.numPorts, where is that code?

To troubleshoot, I used __dict__ to start investigating variable values with the factory and client.  And I used that to start searching through the twisted code in site-packages.  I noticed at least one function which conditionally worked off numPorts in order to shut things down (twisted.internet.protocol.AbstractDatagramProtocol.doStop).  Here's the code:
        assert self.numPorts > 0
        self.numPorts = self.numPorts - 1
        self.transport = None
        if not self.numPorts:
            if self.noisy:
                log.msg("Stopping protocol %s" % self)
            self.stopProtocol()

And so the work around I implemented was to conditionally increase this number while I controlled the disconnect/reconnect.  Seemed to work fine in practice.


Thanks again for the responses.  Given all the benefits Glyph mentioned of ClientService - I've added that migration into my roadmap.

-Chris


-----Original Message-----
From: Twisted-Python <twisted-python-bounces at twistedmatrix.com> On Behalf Of Chris Withers
Sent: Tuesday, March 26, 2019 2:28 AM
To: twisted-python at twistedmatrix.com
Subject: Re: [Twisted-Python] stop/start client connections with loseConnection in ReconnectingClientFactory

On 22/03/2019 23:26, Chris Satterthwaite wrote:
> Hi Chris,
> 
> The files I attached (with the full classes) made it through to my email, but I wondered if they would they go through to everyone.
> 
> Here's a gist with the same scripts:
> https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519
> f
> 
> And yep, as you mentioned, a 'sleep' is definitely blocking.  That's not in the production version; I just dropped it in here for the test script to simulate something.

Observations:

- Your super call at
https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file-testclient-py-L46,
I'd expect that to be super(ServiceClientFactory, self).__init__(), but your spelling may be a python 3 only thing that works?

- Those sleeps are going to cause you more problems than they solve.

- What does this seek to achieve? 
https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file-testclient-py-L88-L89

- Why not use twisted logging instead of print? By setting it to debug, you'll get lots of into about what twisted is doing, and by using logging you won't need to do all that manual traceback printing.

- when you say "When the client hits a problem, it calls transport.loseConnection()", where is that call to transport.loseConnection?

- when you noticed some Twisted source code that works off factory.numPorts, where is that code? Can you provide a link? This doesn't sound right...

Now, as to what your problem is, I suspect it's this call to stopTrying:
https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file-testclient-py-L87

The factory is stopped and started again by ReconnectingClientFactory, so you don't want that there as it means you stop trying every time there's a disconnection.

cheers,

Chris

_______________________________________________
Twisted-Python mailing list
Twisted-Python at twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python