[Twisted-Python] Could Service.startService return a Deferred?

[exarkun at twistedmatrix.com]

On 06:00 pm, peter.westlake at pobox.com wrote:
>On Wed, Nov 27, 2013, at 17:13, exarkun at twistedmatrix.com wrote:
>>On 02:58 pm, peter.westlake at pobox.com wrote:
>>... So would it be possible for
>> >t.a.s.Service.startService
>> >to be allowed to return a Deferred? Then the next service would only 
>>be
>> >started up when the Deferred fired.
>>
>>Probably not.
>>
>>There is some discussion on
>><https://twistedmatrix.com/trac/ticket/5941>.
>
>That's helpful, thanks. I hadn't realized that startService was called
>before the reactor started.
>
>Instead, I've passed in a Deferred to each service that needs to wait.
>How does this look?
>
># -*- mode: python -*-
>from twisted.application import service
>from twisted.internet import defer, reactor
>from twisted.internet.task import deferLater
>
>application = service.Application('Dependencies')
>
>def report(error):
>    print 'ERROR', error
>    reactor.stop()
>
>class Runner(service.Service):
>    def __init__(self, baton):
>        self.baton = baton
>
>    def startService(self):
>        print 'startService', self.name, reactor.running
>        self.baton.addCallback(lambda ignore:deferLater(reactor, 1.0,
>        self.realStartService))
>        self.baton.addErrback(report)

I'd be concerned with the state of `Runner` at this point in the 
process.

What happens if the application gets shut down while that `deferLater` 
is still pending?

`stopService` has a harder job because it might need to deal with a 
service that is partially initialized or a service that is completely 
initialized (and "partially initialized" may cover a multitude of 
different states depending on the complexity of your service).
>
>    def realStartService(self):
>        print 'realStartService', self.name, reactor.running
>
>baton = defer.succeed('pass me to each service')
>
>foo = Runner(baton)
>foo.setName('foo')
>foo.setServiceParent(application)
>
>bar = Runner(baton)
>bar.setName('bar')
>bar.setServiceParent(application)

What about something like this instead?

  @implementer(IService)
  class Runner(object):
      ...
      @classmethod
      def loadFromWhatever(cls, name):
          return deferLater(reactor, Runner, name)

      def __init__(self, name):
          self.name = name

      def startService(self):
          self.running = True
          print 'realStartService', self.name, reactor.running

  def parent(service):
      application.setServiceParent(service)

  loading = Runner.initializeFromWhatever("foo")
  loading.addCallback(parent)
  loading.addCallback(lambda ignored: 
Runner.initializeFromWhatever("bar"))
  loading.addCallback(parent)
  loading.addErrback(stopTheReactorOrWhatever)

The advantage I see of this approach is that a `Runner` never exists in
the service hierarchy until it is fully initialized, started, and 
running.

If a `Runner` is only partially ready and the process shuts down then 
its `stopService` method isn't called because it's not part of the 
service hierarchy.

I could definitely imagine a library to help with this kind of thing. 
For example, perhaps you want the above encapsulated as:

  asynchronouslySetUpServices(application, [
      lambda: Runner.initializeFromWhatever("foo"),
      lambda: Runner.initialifrFromWhatever("bar")])

And maybe then you want to add in some logic so that if the application 
gets shut down while some things are still being initialized then you 
cancel their Deferred.  Then you have good cleanup support for the 
uninitialized case - without complicating `stopService` (the cleanup 
logic is isolated in the implementation of Deferred cancellation where 
it belongs - eg, with this `deferLater`-based asynchronousness it's 
alreay present since deferLater implements cancellation already).

Jean-Paul