[Twisted-Python] Understanding deferred and error handling

[David]

Hi,

I have a hard time figuring out error handling with deferred in twisted. 
More exactly, I don't understand how to always get meaningful tracebacks 
to understand where the error actually happened. For a simple example:

import sys

import twisted.web.client

from twisted.internet import defer
from twisted.internet import reactor
from twisted.python import log

def remote_call():
     # No process bound to 8083 -> connection refused
     d = twisted.web.client.getPage("http://localhost:8083")
     return d

def main():
     d = remote_call()
     def _stop(arg):
         reactor.stop()
     d.addBoth(_stop)

log.startLogging(sys.stdout)
reactor.callWhenRunning(main)
reactor.run()

This will simply print no error in the log:

2011-04-20 12:37:40+0900 [-] Log opened.
2011-04-20 12:37:40+0900 [-] Starting factory <HTTPClientFactory: 
http://localhost:8083>
2011-04-20 12:37:40+0900 [HTTPPageGetter,client] Stopping factory 
<HTTPClientFactory: http://localhost:8083>
2011-04-20 12:37:40+0900 [-] Main loop terminated.

This already bothers me at a fundamental level, because it means it is 
very easy to "swallow" errors without being aware of it. Is there a 
"systematic" solution to this issue, or am I condemned to handle errors 
systematically everywhere in my code ?

Now, if I add an errback for logging purpose:

def main():
     d = remote_call()
     def _stop(arg):
         reactor.stop()
     d.addErrback(log.err)
     d.addBoth(_stop)

I get something like:

2011-04-20 12:38:35+0900 [-] Log opened.
2011-04-20 12:38:35+0900 [-] Starting factory <HTTPClientFactory: 
http://localhost:8083>
2011-04-20 12:38:35+0900 [HTTPPageGetter,client] Unhandled Error
	Traceback (most recent call last):
	Failure: twisted.web.error.Error: 404 Not Found
	
2011-04-20 12:38:35+0900 [HTTPPageGetter,client] Stopping factory 
<HTTPClientFactory: http://localhost:8083>
2011-04-20 12:38:35+0900 [-] Main loop terminated.

I do get an error, but I don't get a traceback. Interestingly enough, if 
I use printTraceback:

def main():
     d = remote_call()
     def _stop(arg):
         reactor.stop()
     def log_error(failure):
         log.err(failure.printTraceback())
         return failure
     d.addErrback(log_error)
     d.addBoth(_stop)

I don't get any traceback either.

I found quite a few documents on how to handle errors in twisted, but I 
did not see much information on how to at least make sure unexpected 
errors are being shown (and fixed later).

cheers,

David