[Twisted-Python] Looking for help dealing with ClientService reconnections

[Daniel Sutcliffe]

Thanks for the response Kevin, a few individual replies embedded below:

On Sat, May 7, 2016 at 3:19 PM, Kevin Conway <kevinjacobconway at gmail.com> wrote:
> I'm still working through your code example and trying to get a better grasp
> of what, exactly, you're trying to implement as far as client behavior is
> concerned. In the meantime, it sounds, on the surface, like you are trying
> to implement a form of the ReconnectingClientFactory:
> http://twistedmatrix.com/documents/current/api/twisted.internet.protocol.ReconnectingClientFactory.html.

pymodbus actually has a pymodbus.client.async.ModbusClientFactory
which is derived from
twisted.internet.protocol.ReconnectingClientFactory
  http://pymodbus.readthedocs.io/en/latest/library/async-client.html

> Your client protocol must be factory aware and call the resetDelay method on
> the factory when a connection is made, but the factory will handle
> reconnecting and generating a new protocol instance. Is that similar to what
> you are trying to accomplish? Maybe see also
> http://twistedmatrix.com/documents/current/core/howto/clients.html#reconnection
> for a quick example of using the reconnecting factory.

I had built working tests using the docs you link to and understand
the way it works, but when Twisted 16.1 came out I decided that as I
was writing fresh code I probably ought to use the suggested new API.
After reading up and checking the source for ClientService it seemed
just what I was after; offering the advantages that Glyph mentioned,
and simplifying the amount of code I had to write (and maintain),
especially as my intention was to use twistd and thus Services.

It was only when I started to implement this that I felt my
inexperience with Twisted's ways of doing things was holding me back
and couldn't see a clear direction of how to build my architecture
around it.

Initially I used my own Protocol class derived from
pymodbus.client.async.ModbusClientProtocol that handled the polling
internally with a Factory that was aware of this class based on
ClientFactory (not ReconnectingClientFactory) that stored the
persistent info (poll rate, etc). This worked well with ClientService
and reconnections happened as expected but once I started to try and
evolve my code to consolidate (from many ClientServices polling
various PLCs) and write out the data (to MySQL) the model felt like
the focus was in the wrong place and that maybe I should be focussing
more on making a derivation of the ClientService have more control of
the polling and just using the Protocol provided by pymodbus as-is.

After hitting the problem I described below, I decided it was time to
ask those with more Twisted experience what the most appropriate way
to approach this problem was. Am I better off going back to doing
polling in Protocol with a Factory that provides ways to pump the data
back upstream to where it needs to be consolidated, and not rolling my
own ClientService to control it? or do I just need a better way of
getting my ClientService to be aware of the latest connection? Maybe
there's a pathway in between, or even something I haven't yet thought
of?

If anyone does have the time and desire to help me with this but feels
they need more info from me or background then please just ask, and I
will provide and be most appreciative :)

Cheers
/dan

On Fri, May 6, 2016, 12:23 Daniel Sutcliffe <dansut at gmail.com> wrote:
>> Hello all, my first post here - only been using Twisted for about a
>> month and am also a relative newcomer to Python but have been coding
>> professionally for 20+ years. I was attracted to Twisted and Python
>> for a particular project purely because after research it seemed to be
>> the best tool for the job, and have actually been enjoying both Python
>> and Twisted much more than I ever thought I would.
>>
>> The project I am coding towards is creating a sensor data collection
>> gateway. First iteration needs are simply pulling data from ModBus TCP
>> slave PLCs and writing it to a MySQL database, but goals beyond that
>> are making the source of the data and its destination(s) very
>> flexible(pluggable). Therefore I am trying to create a good clean
>> architecture from the outset so as I iterate forwards I don't finish
>> up having to take too many steps backwards before heading forwards.
>>
>> I am using pymodbus to pull the data which works well for my devices,
>> has a twisted async API, and have created more than a few prototypes
>> that demonstrate all works as I expect. Where I am a bit stalled is
>> getting to grips with a good architecture that fulfills my needs - my
>> intention is that the application that meets my first goal will be a
>> twistd plugin.
>>
>> The new ClientService class seems like it will fit my needs very
>> closely but I am struggling with how to handle the reconnections... I
>> have been using the whenConnected() method to grab the Protocol for
>> the initial connection and then use a method of this to poll the
>> connected slave. When the connection is lost I get an errback from
>> this method's deferred which I use as a signal to abandon the Protocol
>> and call whenConnected() again... at this point I have an issue though
>> as the returned deferred immediately gives me a callback with the same
>> Protocol which has just lost its connection, and thus loop...
>>
>> Before I got on this mailing list I posted this Q to stackoverflow
>> with some example code:
>>     http://stackoverflow.com/q/37061807/3448214
>> but no solution or much attention there yet.
>>
>> As I say there, I realize I have probably just made a bad pattern
>> choice for how to use this API, but I have not been able to work out a
>> better choice which seems clean and fits my needs/understanding well.
>> I have tried deriving my own Protocol/Factory and handling the polling
>> there but this seems to get really messy once I start to add code to
>> get the collected data to a destination at that level, involving
>> giving the Protocol too much knowledge of how the data is to be
>> handled.
>>
>> Any advice, good patterns, or pointers to other projects which do
>> something similar is appreciated,
-- 
Daniel Sutcliffe <dansut at gmail.com>