While the specific optimization in question might not work, there's definitely something we can do to improve this situation.
Brian Warner did a really good writeup of the implications of this problem in a ticket on the allmydata tracker which he later copied as a message to the Twisted list.
However, the suggested fix there (reactor.eventually()) is wrong, in my opinion. You don't need to rip the whole stack in order to fix this problem, only the portion that is already running inside defer.py.
Deferreds are good at blowing the stack
Returning a Deferred from the callback of another Deferred too many times results a RecursionError
Site design credits