Opened 4 years ago

Last modified 3 years ago

#5941 enhancement new

facilitate dependency between service state at startService time

Reported by: teratorn Owned by:
Priority: normal Milestone:
Component: core Keywords:
Cc: Branch:


It is often desirable to initiate a controlled, asynchronous start-up of a Service, or MultiService collection. E.g. a database may need to be contacted before the Service can be considered "started".

To accommodate this, startService() and privilegedStartService() could be documented as returning None, or a Deferred.

Twisted code that consumes Service objects should be enhanced to support the extended interface, and chain the Deferred, or otherwise take an appropriate action to propogate the startup result.

By default, I think MultiService.startService()'s Deferred should errback if any of the child Service objects errback, as consistent with the current case of synchronous startup (and synchronous Python exceptions).


Change History (5)

comment:1 Changed 4 years ago by Jean-Paul Calderone

Services are started before the reactor is running. It's not possible to have asynchronous operations complete until the reactor is running, so it's not possible for service startup to be asynchronous.

As far as the semantics of whether a service is "started" or not before it has, eg, established a database connection go, this isn't very interesting. I could argue that as soon as the service attempts to connect to the database, it has started. It may not be usable, but "usable" and "started" sound different to me. However, as I said, this isn't very interesting, since it's just a game with words. "started" has a meaning currently, and it is the meaning it currently has, not the meaning proposed on this ticket. :)

Also, privileges cannot be shed until all privilegedStartService implementations have run. This means that being able to return a Deferred from privilegedStartService vastly increased the amount of code you're running as a privileged user. This is a bad thing.

I suggest that what you're really after is an idiom for asynchronously initializing a service hierarchy as different parts of it become ready for use. This may merit better documentation than is currently present (really, what doesn't), but it's a fairly straightforward thing. For example,

    d = DatabaseService.fromConnectionString(description)
    d.addCallback(lambda svc: svc.setServiceParent(application))

There, now you have a service hierarchy which does not contain incompletely initialized objects. Moreover, if you also chain the services which depend on DatabaseService on that initialization Deferred, you'll never find yourself with code that tries to access a not-yet-initialized service.

I'll admit that the one shortcoming here is the case where the application is stopped before all initialization has been started. For this, it really might be nice if the service hierarchy knew about the pending services so it could cancel them if they did not initialize fully before someone decided to stop the service. This probably has nothing to do with returning Deferreds from startService, though.

comment:2 Changed 4 years ago by Glyph

Summary: Support asynchronous Service startup-up - startService() may return a Deferredfacilitate dependency between service state at startService time

The feature being requested here is dependency between services, not "asynchronous" service start-up. Service start-up is by some definition (as exarkun notes above) never asynchronous, because it has to happen pre-reactor-startup. Service startup is by some other definition always asynchronous, because you have to fire off all your Deferreds in parallel.

Returning Deferreds from startService is a particularly bad way to handle this, because this would encourage services which are only even potentially dependent to start up in serial, as slowly as possible (mirroring the principal design error of SysV init, which every single platform in the universe has at least 2 attempts to correct).

What you really want is a way to communicate some required state to a service which is starting up. Since not having required state around at the time you construct the service is bad (you don't want to have incompletely-initialized objects floating around that have to be mutated before they can do anything), you really want to construct your dependent service at the time that it is ready to come into being.

Calendar Server has developed an idiom for doing this, although I'll be the first to admit that it leaves many things to be desired. However, it is a reasonable example of the use-cases that a large, real-world Twisted application will encounter, if it has many dependent services and phases of startup which may or may not occur based on runtime state or application configuration.

comment:3 Changed 4 years ago by Tom Prince

I was looking for info on this recently, and found a couple of related threads. Recording them here, for anybody working on this in the future.

comment:4 Changed 3 years ago by Matt

startService is called before the reactor starts. But what about stopService? Could that be changed to return a Deferred? Perhaps a service connects to the database, and once there's a connection its job is done and can be stopped as a signal to other services to start.

comment:5 Changed 3 years ago by Glyph

Note: See TracTickets for help on using tickets.