Add timeout support to xmlrpc.Proxy.callRemote

[abierbaum]

I have been using xmlrpc successfully but I have run into an error state recently where a remote call just hangs indefinitely. It seems that the server is simply not replying to the call. It would be very nice to have a timeout parameter for callRemote or Proxy that would call the errback after a given amount of time and close down the remote call request.

We discussed this on IRC and here is the relevant conversation:

<allenb1> hello all.  I am rather new to twisted and I am running into a problem with xmlrpc.Proxy.callRemote.  When I make a call the communication seems to hang.  I was wondering if there was a way to set a timeout on the remote call or directly on the deferred.
<allenb1> Any ideas?
<itamarst> there may be a timeout keyword argument
<itamarst> not sure if that's timeout on response though...
<exarkun> You can use reactor.callLater to set up an application-level timeout.
<allenb1> checking the source at: http://twistedmatrix.com/trac/browser/trunk/twisted/web/xmlrpc.py ....
<allenb1> application-level:  I thought about that but what I am seeing is that if one call hangs then future calls all seem to hang.  This could just be a side-effect of how my application is structured though.  I would need to test it in more detail.
<allenb1> I don't see anything about a timeout in the xmlrpc.py file.
<allenb1> It looks like it is asking the reactor to connectTCP using an internal _QueryFactory class.  Is there some way to add a timeout to either the factor or the connection call?
<itamarst> well, if http client class inherets from TimeoutMixin it should be fairly easy (and might even be on by default) 
<itamarst> if not, you may want to file a bug
<itamarst> future hangs calling is not likely to be twisted's fault as such
<itamarst> except in so far as the Deferred is never triggering
<allenb1> itamarst: I don't see http.HTTPClient deriving from TimeoutMixin, but like I said I am rather new to twisted.
<exarkun> The client doesn't, indeed.
<allenb1> To my newbie eye's it looks like this is currently unsupported and is probably a feature request unless I am missing some magic somewhere.
<exarkun> It looks unsupported to me as well.

[cprov]

I've run into the same problem today and prepared an example solving it.

 http://pastebin.com/f4105f1de

I works with the documentation server available in  http://twistedmatrix.com/projects/web/documentation/howto/xmlrpc.html

[mwh]

To make this more complicated, the stuff about TimeoutMixin only handles the case when the server accepts the connection but does not respond in time. It doesn't help with the server does not accept the connection -- it seems like to do this you need to pass a timeout to the reactor.connectXXX call in twisted.web.xmlrpc.Proxy.callRemote.

And it seems like it's going to be hard to deterministically test this timeout :(

[cprov]

Replying to mwh:

To make this more complicated, the stuff about TimeoutMixin only handles the case when the server accepts the connection but does not respond in time. It doesn't help with the server does not accept the connection -- it seems like to do this you need to pass a timeout to the reactor.connectXXX call in twisted.web.xmlrpc.Proxy.callRemote.

Hi Michael, reactor.connectXXX() timeout defaults to 30 (seconds) which isn't bad at all, it will fire the errback with error.TimeoutError if the connection couldn't be established within this period.

And it seems like it's going to be hard to deterministically test this timeout :(

If we don't extend callRemote() we can rely on twisted testsuite.

BTW, the TimeoutMixIn fires the errback with error.ConnectionDone, it's all detectable in the call sites.

Can thank you guys enough for such a great job done on twisted! :)

[exarkun]

It's now possible to implement an overall request timeout externally to Proxy. The Deferred returned by callRemote is now cancelable.