[Twisted-Python] What to do when a service fails to start, also, deferred and startService

glyph at divmod.com glyph at divmod.com
Sun Dec 21 03:20:32 EST 2008


On 28 Nov, 03:38 pm, terry at jon.es wrote:
>Hi Glyph
>
>Thanks for the detailed reply.

No problem, sorry it took so long to get back to this; I definitely left 
the conversation halfway through.
>>>>>>"glyph" == glyph  <glyph at divmod.com> writes:
>glyph> On 27 Nov, 05:16 pm, terry at jon.es wrote:

>glyph> The key question here is: indicate to whom?  If you want to 
>indicate
>glyph> it to some other object, well, try:except: or addErrback and 
>call a
>glyph> method on that object.  Nothing magic about it.
>
>I have code written as a Twisted plugin. So I have a class implementing
>IServiceMaker and IPlugin, and I create an instance of that class which
>gets found when I invoke twistd from the command line.
>
>So in my case I want to indicate to twistd that the service that my 
>class
>creates a makeService method to create, but which I do not set in 
>motion,
>has failed to start and that twistd should exit, or do something other 
>than
>cheerfully tell me that there's been an Unhandled Error.

>Does that make more sense? Sorry, I should have said I was using 
>twistd.

Yes.  And I think that this is a good use-case for making IService a bit 
deeper than it is.  twistd (and, one day, other tools that might be able 
to start IService objects) potentially needs some more information so it 
can make a more intelligent high-level decision about what to do next.

My previous messages were basically saying "you can't do what you want 
with IService".  i.e. you can't write your own code to do something 
clever and somehow have dependencies and notifications to users of 
"twistd" fall out of that.  But that shouldn't be taken as an indication 
that twistd itself shouldn't be improved.

For reasons I brought up in previous messages, exiting is not always the 
right thing to do.  But in some cases (*none* of the services on offer 
were able to start up, for example) it might be.
>I'm not sure what should happen. I'm sitting at the command line, I've
>asked twistd to start something for me, there's clearly been a problem
>doing so (and this doesn't have to be baroque, maybe I just couldn't 
>listen
>on a specific port I wanted, or maybe my code somehow raised an 
>Exception),
>but I don't seem to have a mechanism for having twistd take any notice 
>at
>all.

Keep in mind that you may be sitting on the other end of a blackberry 
which needs to get an email when 'twistd' fails to start up after a 
server reboot - you're not necessarily at a command line.  One of the 
reasons that no mechanism has yet evolved for propagating interesting 
status messages about services is that there's no obvious channel for 
those messages to be communicated to.
>In the case of a service being started by twistd, it doesn't seem as 
>simple
>as you describe, but maybe that's my lack of understanding again.
>I can easily subclass IService, but something else is calling the 
>startService
>method of that subclass. And that thing, whatever it is, is not 
>expecting
>me to return a deferred. So if my startService has for some reason got 
>its
>hands on a deferred, it can't simply hand it back to its caller and 
>have
>something (twistd in my case) see that an error occurred.

>It does feel like I have to track down what this something else might
>be.

It's not that you've failed to understand something - you have correctly 
identified that there is nothing to understand :).  You need to go find 
that thing ("whatever it is" ;-)) in twistd that's invoking the very 
first call to IService.startService (and privilegedStartService as well) 
and propose a concrete way to make it smarter.

(I've elided your exploration of twisted code, but you're basically 
looking in the right direction.)
>So should I write my own twistd?

No way, man, it's open source!  Not writing your own is the whole point! 
Open a ticket, write a patch.  If you're not really sure what the patch 
should do, we can continue this discussion about reporting startup 
errors and service dependencies here.  This poorly-specified ticket 
indicates that other developers have had similar problems in the past: 
http://twistedmatrix.com/trac/ticket/1572
>All this doesn't seem to be a matter of
>simple subclassing. Plus, I can't just go in and start editing the
>top-level functions in twisted/application/app.py and
>twisted/scripts/_twistd_unix.py or code that imports app, etc.

Sure you can.  Then, you take the results of running 'svn diff' on the 
copy of Twisted where you did that, and... :)
>Sorry for so many questions - I really don't know if I'm missing 
>something
>simple here. I do enjoy digging into all this, and I appreciate your
>apparently limitless patience. I wish I knew it all better. Twisted is
>complex and it's a pretty good bet that anything you think of or run 
>into
>as a n00b has been thought of or encountered before, and that whatever 
>way
>you think of to solve it will probably be non-optimal, or plain wrong, 
>or
>in ignorance of a solution someone much more experienced has already
>implemented, or... etc.  Hence my many questions.

Unfortunately it's equally likely that it's come up a million times and 
a half dozen people know it's an unresolved issue but it's never been 
filed as a ticket and never been clearly framed as a specific problem 
rather than a vague architectural queasiness.  The value of a good 
ticket should not be underestimated.
>In fact I have something like a process pool running in one service and 
>I
>talk to it from another machine. I say "hey, process pool, start me up 
>the
>following service (a twistd service)" and I would then like to know if 
>that
>service started, and if not then why not. So having twistd fail or 
>report
>an error if it can't start a service would be useful.

Quite so.  But, this is a great example of how this is a use-case beyond 
twistd's current capabilities, but not out of the scope of its eventual 
ambition.  It isn't currently designed to manage processes on remote 
machines.  Maybe ampoule has something which does?  I don't know, that's 
a tricky area which requires a lot of thought beyond this one feature.
>glyph> Doing either of those things would definitely be wrong.  There's 
>no
>glyph> reason to sys.exit or reactor.stop if your application can't 
>start
>glyph> up, unless your management system specifically calls for such a
>glyph> thing.
>
>Maybe this is a case where it's (semi-)justified. At least if I called
>sys.exit the twistd process would go away, instead of sitting there 
>acting
>as though nothing's wrong :-) I can also, of course, try interacting 
>with
>the service I think I just started on the remote machine, and if I 
>can't
>then I can tell the original process pool to kill the twistd process. 
>But
>that seems a pretty roundabout alternative to just having twistd notice
>that something went awry when calling startService.

It may be worthwhile, before writing a patch, to devise your *own* 
error-reporting channel on top of twistd.  For example, a dedicated 
error-report-processing server listening on a dedicated port.  Then you 
can specify your own enhanced-IService interface that handles 
dependencies and deferreds and whatever else you need, and an adapter to 
hooks it up to twistd via IService but reports errors to your external 
error-reporting channel.  Once you're sure that design works, it should 
be straightforward to change the implementation of twistd to honor your 
expanded interface, and provide an alternate implementation of the 
error-reporting channel.
>glyph> In the future, even the Twisted plugin code might be starting 
>some
>glyph> things in addition to your application.  As I mentioned above, a
>glyph> good reason to do that is to perform diagnostics on failed 
>startups
>glyph> :).
>
>I assume you really mean "startups" and not "services". In which case, 
>I'm
>100% sure there's something funny here, but I can't figure it out. I'd 
>love
>to know, and I'm smiling broadly in any case. Too bad email loses so 
>much
>humor.....  we can always try though.

By "startups" I simply meant "attempts to start-up a service" i.e. calls 
to startService.  I'm sure there is a truckload of subtle and completely 
unintentional humor in there somewhere though.




More information about the Twisted-Python mailing list