[Twisted-Python] Remote PB calls in ApplicationService.stopService

[Brian Warner]

Alex Levy <mesozoic at polynode.com> writes:
> 
>    def stopService(self):
>      somePBRemoteCall(i_am_dying_event)
>      app.ApplicationService.stopService(self)
> 
> On the sending side, this results in 'Connection failed'. On the receiving
> side, the following traceback appears. Could somebody more familiar with the
> internals of Perspective Broker explain what's going wrong, or how I might
> change my approach?

There are a couple of problems interacting here. One is that you're sending
out data as the application is shutting down, which means that many of the
usual assumptions (about the request eventually being sent over the wire,
about a response eventually coming back) are not guaranteed to remain true.

A second is that, currently, all PB calls require acknowledgement, even if
you expect the answer to be a simple 'None' (like the None that's returned
implicitly when you fall off the end of a Python function). There are hooks
in place to allow a call to tell the receiver that no answer is expected (or
will be accepted), allowing that end to skip the response phase, but those
hooks are disabled because there is no way to the caller to know whether
you're going to add a callback to the Deferred or not (especially because
that callback could be added *after* the request has been sent). So for now,
the far end always acknowleges every single PB call.

Or at least it tries to. I suspect you're seeing the following sequence of
events:

 reactor.stop()
  'shutdown' system event trigger is fired, callbacks are run
    app._beforeShutDown is run as a 'before' trigger on the 'shutdown' event
     stopService() is run on all Services
      callRemote(aaaaaarrgh)
       remote_aaaaaarrgh is serialized and queued, maybe sent, maybe not
    app._afterShutDown is run as the 'after' trigger, does app.save()
 reactor exits
 program exits
 all sockets are closed

The serialized remote call may or may not get sent: if the transport's
.write method tries to write some data immediately, then it will probably
get out. But the connection will be dropped soon afterwards, probably before
the far end has completed sending an answer.

Given that PB is about acknowleged method invocations, it will probably be
cleaner to try to delay shutdown briefly while you get your "I am dying"
messages through. Fortunately, both .stopService and the before-shutdown
event trigger that drives it have a useful property: you can return a
Deferred from them and the rest of the shutdown process will be stalled
until that Deferred is fired.

So just do this:

   def stopService(self):
     d = somePBRemoteCall(i_am_dying_event)
     app.ApplicationService.stopService(self)
     return d

(or do the superclass call first, it doesn't matter)

This will send out your shutdown messages but then return to the main
reactor loop, waiting for the messages to be acknowleged before proceeding
with the rest of shutdown (app.save() and exiting the reactor).

BTW: I think the exception you're seeing is a bug, and results from the
connectionLost messages being invoked twice on the same Broker. I think this
is an edge case that can only happen when a sender does something weird.

hope that helps,
 -Brian