[Twisted-Python] Looking for help dealing with ClientService reconnections

[Glyph]

> On May 10, 2016, at 9:52 AM, Daniel Sutcliffe <dansut at gmail.com> wrote:
> 
> Thanks Glyph, I think you have given me a push back in the 'right'
> direction - more thoughts and commentary embedded below if you, or
> anyone else, has the time.
> 
> On May 6, 2016, at 10:19 AM, Daniel Sutcliffe <dansut at gmail.com> wrote:
> [...]
>>> The new ClientService class seems like it will fit my needs very
>>> closely but I am struggling with how to handle the reconnections... I
>>> have been using the whenConnected() method to grab the Protocol for
>>> the initial connection and then use a method of this to poll the
>>> connected slave. When the connection is lost I get an errback from
>>> this method's deferred which I use as a signal to abandon the Protocol
>>> and call whenConnected() again... at this point I have an issue though
>>> as the returned deferred immediately gives me a callback with the same
>>> Protocol which has just lost its connection, and thus loop...
> 
> On Mon, May 9, 2016 at 6:19 PM, Glyph <glyph at twistedmatrix.com> wrote:
>> If you want a hook each time a new protocol is created, you're probably
>> better off writing a wrapper protocol factory, and passing that to your
>> ClientService, then doing any set-up work you want to do in your
>> buildProtocol implementation, which delegates to the real, pymodbus
>> implementation.
> 
> Understood, if this is the way the framework is intended to be used I
> realize doing anything else is going to be fighting against the flow.
> 
> However, just to probe the situation I found myself in further, for
> the sake probing broken code to see how it might be fixed:
> 
> Given the ClientService.whenConnected() method is intended to provide
> access to my connected Protocol through the deferred it returns, is it
> not a little unfriendly that this Protocol may turn out to be
> disconnected? OK occasionally due to timing but for this to be a
> possible condition which can loop with the same disconnected Protocol
> returned until the ClientService has its _currentConnection set to
> None, suggests to me that I can't safely use my Protocol from
> whenConnected() for much other than as a notification the first
> connection has occurred... but how do I avoid this?

whenConnected() is not intended to be used for "give me each Protocol as it is instantiated so that state can be set up", it is intended for API clients which want to send a message to the current connection to just retrieve the current connection so they can call a method on it.

I'm not sure what you mean by "turn out to be disconnected".  The physical reality of networking is that you might always encounter a transport which has been disconnected but which you haven't received notification of its disconnection yet.

> I have looked at the source and it seems to me the fact that the
> connection has been lost should bubble up to the ClientService through
> a t.a.i._DisconnectFactory and t.a.i._ReconnectingProtocolProxy once
> my Protocol's connectionLost() is called. My issue seems to be that I
> errback on a Protocol method's deferred returned to code at or above
> the ClientService level which gives up on that Protocol and calls
> whenConnected() to get the next one, only the Protocol's
> connectionLost() has yet to be called and then doesn't have chance to
> because my code is looping around calling whenConnected() and getting
> the same Protocol back. I hope that makes sense :-/

Let me try to rephrase: you call a protocol method which returns a Deferred; you add an errback to that Deferred which calls whenConnected() to re-try, but since the protocol hasn't disconnected yet, you get the same protocol instance back, which is useless to you.

> My Q on this is if I should be internally calling my Protocol's
> connectionLost() so it can bubble up to the ClientService before I
> errback on the Protocol method - whose responsibility is it to call
> this?

It's the framework's responsibility to call it.  You should not call it yourself.  Your Protocol's connectionLost isn't going to bubble up to ClientService anyway; you'd have to call your wrapper's connectionLost, which would confuse its internal state, since the framework would call it again right afterwards, and we definitely don't have test coverage for that, since the framework will normally only call it once.

The right way to handle this would be to introduce a delay between re-tries.  It's generally a good idea to have such a delay for lots of reasons; you don't want to overload your peer in the case of a transient failure.  As a bonus, the fact that you've gone back up to the reactor loop to wait a while means that the transport will be properly disconnected and whenConnected() will do what you want.

>>> Before I got on this mailing list I posted this Q to stackoverflow
>>> with some example code:
>>>   http://stackoverflow.com/q/37061807/3448214
>>> but no solution or much attention there yet.
>>> 
>>> As I say there, I realize I have probably just made a bad pattern
>>> choice for how to use this API, but I have not been able to work out a
>>> better choice which seems clean and fits my needs/understanding well.
>>> I have tried deriving my own Protocol/Factory and handling the polling
>>> there but this seems to get really messy once I start to add code to
>>> get the collected data to a destination at that level, involving
>>> giving the Protocol too much knowledge of how the data is to be
>>> handled.
>> 
>> I am curious as to why you say that this is "messy".
> 
> Honestly, this was just a gut feeling at the time, probably more
> sourced in my implementation from lack of experience in using Twisted;
> after reading around the subject, looking at many more examples, and
> your advice, I think I am convinced I need to back to looking at my
> own Protocol derived from the pymodbus one with a Factory that
> contains the persistent config and access to an interface to pump the
> polled data upstream.
> 
>>> Any advice, good patterns, or pointers to other projects which do
>>> something similar is appreciated,
>> 
>> I spent a while thinking about your question, and I'm sorry that I can't give
>> a more thorough answer, but I think you need to be a bit more specific
>> about what it is you don't like about your potential solution.  It seems to me
>> that having a delegating Factory, especially if all you need to do is set up
>> some state on each Protocol that gets produced, should be sufficient...
> 
> Looking at the code again I think it just seemed to make sense to me
> at the time to have something that is (or has) a ClientService be the
> object I am calling a DataSource have more control over when it polls
> that data and what it does with it - and not to have to make the
> Protocol and its Factory aware of this at all. Does that make it any
> clearer? I am happy to push ahead with building this into my
> Factory/Protocol if that is more normal usage for Twisted as I am sure
> there will be benefits of encapsulating it here beyond the other side
> I was seeing during that moment of confusion.
> 
> Thanks for the advice, I think it was enough to nudge me in a
> direction that will work better to get me going, and if with more
> experience other usage makes more sense I can always refactor,
> refactor, refactor :)

Yup!

Happy to help, sorry for the super long lag time on this reply, but my email queue has been pretty full lately :)

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20160521/52b6366b/attachment-0002.html>