[Twisted-Python] Looking for help dealing with ClientService reconnections

[Daniel Sutcliffe]

Delayed response here but hopefully it is still seen as worthwhile
discussion, sorry if it is seems to just be beating a dead horse ;)

On May 10, 2016, at 9:52 AM, Daniel Sutcliffe <dansut at gmail.com> wrote:
> > Given the ClientService.whenConnected() method is intended to provide
> > access to my connected Protocol through the deferred it returns, is it
> > not a little unfriendly that this Protocol may turn out to be
> > disconnected? OK occasionally due to timing but for this to be a
> > possible condition which can loop with the same disconnected Protocol
> > returned until the ClientService has its _currentConnection set to
> > None, suggests to me that I can't safely use my Protocol from
> > whenConnected() for much other than as a notification the first
> > connection has occurred... but how do I avoid this?

> whenConnected() is not intended to be used for "give me each Protocol as it
> is instantiated so that state can be set up", it is intended for API clients
> which want to send a message to the current connection to just retrieve the
> current connection so they can call a method on it.

I appreciate now this 'protocol/connection init' isn't what the
whenConnected() method was intended for, and it wasn't what I was
using it for when I came across my issue. However there seems to me to
be no documentary discouragement for using it this way... at least for
someone who is not as familiar with Twisted's common usage patterns

> I'm not sure what you mean by "turn out to be disconnected".  The physical
> reality of networking is that you might always encounter a transport which
> has been disconnected but which you haven't received notification of its
> disconnection yet.

The 'turns out to be disconnected' actually came from my acceptance
that network connections drop/fail and I have little control of how or
when this happens. Thus I may call a method on a Protocol at any time
only to find that it is not in a good state to handle my request.

My (misguided) goal was to call a method of the Protocol returned from
whenConnected() every so often to give it a task to do; my expectation
being that *when* the connection failed I would find out from the task
method's returned deferred and could then just fire off another
ClientService whenConnected() to get the next useful Protocol once it
is established, and then use this to continue giving the Protocol
tasks until it also ultimately fails...

> > I have looked at the source and it seems to me the fact that the
> > connection has been lost should bubble up to the ClientService through
> > a t.a.i._DisconnectFactory and t.a.i._ReconnectingProtocolProxy once
> > my Protocol's connectionLost() is called. My issue seems to be that I
> > errback on a Protocol method's deferred returned to code at or above
> > the ClientService level which gives up on that Protocol and calls
> > whenConnected() to get the next one, only the Protocol's
> > connectionLost() has yet to be called and then doesn't have chance to
> > because my code is looping around calling whenConnected() and getting
> > the same Protocol back. I hope that makes sense :-/
>
> Let me try to rephrase: you call a protocol method which returns a Deferred;
> you add an errback to that Deferred which calls whenConnected() to re-try,
> but since the protocol hasn't disconnected yet, you get the same protocol
> instance back, which is useless to you.

Thanks Glyph, that seems like exactly what I was trying to say.

> > My Q on this is if I should be internally calling my Protocol's
> > connectionLost() so it can bubble up to the ClientService before I
> > errback on the Protocol method - whose responsibility is it to call
> > this?
>
> It's the framework's responsibility to call it.  You should not call it
> yourself.  Your Protocol's connectionLost isn't going to bubble up to
> ClientService anyway; you'd have to call your wrapper's connectionLost,
> which would confuse its internal state, since the framework would call it
> again right afterwards, and we definitely don't have test coverage for that,
> since the framework will normally only call it once.

I'm glad that I got this mostly right in my head - it didn't feel
right for me to be calling this, or any similar method I could find.

> The right way to handle this would be to introduce a delay between re-tries.
> It's generally a good idea to have such a delay for lots of reasons; you
> don't want to overload your peer in the case of a transient failure.  As a
> bonus, the fact that you've gone back up to the reactor loop to wait a while
> means that the transport will be properly disconnected and whenConnected()
> will do what you want.

I didn't even consider this as my thinking was along the lines of the
ClientService having the ability to handle the connection re-tries and
delays to avoid any overloading of the service it wraps.

It would be nice if there was something I could do in my Protocol task
method such that it didn't fire the errback on its deferred until its
Factory (and ultimately its Service in this case) had been made aware
the connection had failed. Does this not have to be done elsewhere in
Twisted or other projects using Twisted, or is the common pattern just
to introduce a delay to avoid any possible race condition?

My current intention is to have the Protocol's Factory handle the
assigning of tasks to its Protocol which seems the 'more normal'
direction to take, so any discussions above are purely for my
education and intellectual curiosity - I'm glad of any and all
feedback.

Cheers
/dan
-- 
Daniel Sutcliffe <dansut at gmail.com>