[Twisted-Python] What to do when a service fails to start, also, deferred and startService
glyph at divmod.com
glyph at divmod.com
Fri Nov 28 09:02:17 EST 2008
On 27 Nov, 05:16 pm, terry at jon.es wrote:
>I imagine that it must be common that people write services that don't
>simply launch things listening on sockets, but instead need to do a
>of things, one after another, in order to get going and to be ready to
>provide their service (or multiservice).
I can't speak to how common it is, but I don't do it and I've actually
seen it fairly rarely; although I have heard people asking about it a
number of times.
For me, baroque and elaborate start-up dances are a code smell.
Services should be as independent as possible. Of course, sometimes
some kind of initialization conversation is unavoidable, but I do like
to try to keep it as short as possible.
>If you do need to write something like that, it seems the chances are
>pretty high you're going to be calling code somewhere along the way
>returns a deferred. And because the twisted/application/service.py code
>that calls startService doesn't handle deferreds being returned, this
>creates a real problem. At least as far as I understand things - which,
>usual, may not be very far.
I think you're misunderstanding what a "service" is. The word is,
perhaps, a bit to lofty for its humble job. A service is just an event
notification mechanism that tells you when it's time to start up, and
when it's time to shut down.
I can understand why it would be attractive to misunderstand in this
way, though: IService doesn't do very much, you have requirements that
it doesn't cover, and if it were the thing you understand it to be then
it would cover those requirements. I'm sure that would be nicer for you
This might seem a bit inconsistent, since stopService uses the return of
a Deferred. However, this is for a very specific reason, not a
generalized error-handling case: you may need to prevent the *rest* of
the system (specifically, the reactor) from completely shutting down
until you've managed to cleanly shut down whatever you're trying to shut
down on potentially remote systems. startService has no such problem
though; the service subsystem has told you "It's time to start up!" -
its job is done, and the reactor isn't going away as part of service
startup, so it's your responsibility as an application author to make
sure your other dependencies are properly initialized.
>But if something does go wrong, you've got a failure propagating its
>down a errback chain, eventually (unless an errback switches you back
>the callback chain) popping out the end and causing the reactor to
>Unhandled Error message. So you can't indicate that the service has
>to start by throwing, because the exception is going to pop harmlessly
>the end of the deferred chain as a generic unhandled error and will not
>cause Twisted to know that the service couldn't start.
The key question here is: indicate to whom? If you want to indicate it
to some other object, well, try:except: or addErrback and call a method
on that object. Nothing magic about it.
There is no general-purpose object in Twisted who would be interested in
any and all kinds of failures. Except, of course, the logging system,
which, as you say, has already been told about this.
>This all feels quite ironic :-) Twisted leads you coyly into the dark
>powerful world of working with and heavily depending on Deferreds. But
>then, right when you expect it to be there for you, covering your back,
>throws up its hands as if to say "What!!? You expect me to deal with
>returning a Deferred? You gotta be kidding, sucker."
This begs the question, again, of what does it mean to "deal with"
returning a Deferred? Pause the service startup chain? As exarkun
noted in the thread you referenced, we *can't* stop and do that in
privilegedStartService, so it would be a bit asymmetric to do so in
In what way would you expect the service mechanism to "deal with"
returning a Deferred? Stop starting other services? Print out some
different log message?
The options I can come up with are generally undesirable. Service order
is somewhat arbitrary. If you have a debugging service (like manhole)
that happens to start up after your failed startup, then you won't be
able to log in and inspect your failed-to-start service if it fails. If
you amend the log message in some way, chances are good that you will
remove information (stack frames) that would be useful for debugging.
These are equally good reasons not to pause the service startup chain in
MultiService, too: one service should be able to inspect another to see
why it's hung.
>I could follow Moof's approach (last poster in the above thread), but
>seems to just pass the problem on to a higher level, where something
>is calling startService (or something earlier) and so on up until we
>the topmost point at which something is not allowing/expecting a
>to come back. Should I track down and subclass all these things? That
>would seem cruel and unusual punishment to the faithful Deferred user,
>having to go in and subclass core classes because they don't deal with
Indeed. This problem is left to a higher level because it is a higher
level problem. There is certainly a case to be made that the higher
level should be somewhere in Twisted itself, but let's not complicate
IService is a very, very simple interface. If you want to respond to
failures from startService (deferred failures, exceptions, or whatever
else) in a useful way, then you can write your own implementation of it
which manages startup order, keeps track of dependencies, and maintains
a state machine that handles stopService appropriately if called in mid-
I don't think that having to implement an interface with 6 methods on it
could be considered "cruel and unusual". If you think so you may want
to investigate options other than Twisted: you will frequently be
expected to implement interfaces with methods on them ;-).
There's no need to "track down and subclass" lots of things. Your
IService wants the things that it contains to have a richer interface
which allows for error handling, dependencies, and propagation, so
simply write a single wrapper for simpler IService objects that expands
the interface to do the other things that you're interested in.
This all strikes me as totally straightforward and easy, and I don't
think I'm any kind of super-genius for being able to write a few Python
classes that call a few simple start/stop methods in the order that I
want them to run in :).
>I could do something dramatic, like call reactor.stop or sys.exit in my
>errback chain, but those seem completely wrong. Apart from the
>possibility that something other than Twisted plugin code is trying to
>start my service, it's also anachronistic because it will happen at
>unpredictable time after startService has gotten back (and ignored) the
>deferred and Twisted has moved on (perhaps even to start other
Doing either of those things would definitely be wrong. There's no
reason to sys.exit or reactor.stop if your application can't start up,
unless your management system specifically calls for such a thing. In
the future, even the Twisted plugin code might be starting some things
in addition to your application. As I mentioned above, a good reason to
do that is to perform diagnostics on failed startups :).
More information about the Twisted-Python