[Twisted-web] [Twisted-Python] Speed of rendering?

[Peter Westlake]

On Wed, Oct 24, 2012, at 09:16, Glyph wrote:
> 
> On Oct 23, 2012, at 8:10 AM, Peter Westlake <peter.westlake at pobox.com>
> wrote:
... 
> > The problem I'm having is that flatten() returns immediately if given a string or anything else without an unfired Deferred, and that sends Client._continue into an unbounded recursion. Is there a general good way to handle this kind of problem? Somehow I need to return control to the reactor long enough for Client._request to return.
> 
> That sounds like a bug, although it's hard to say without seeing the
> exact code that you're talking about.  Can you send a representative
> example?

Here it is:


from benchlib import driver, Client

from twisted.web.template import flatten
from twisted.web.server import Request
from twisted.web.http import HTTPChannel

class Client(Client):
    channel = HTTPChannel()
    request = Request(channel, False)

    def _request(self):
        d = flatten(self.request, 'hello', lambda _: None)
        d.addCallback(self._continue)  ### Infinite recursion happens
        here
        d.addErrback(self._stop)


def main(reactor, duration):
    concurrency = 1
    client = Client(reactor)
    d = client.run(concurrency, duration)
    return d


if __name__ == '__main__':
    import sys
    import flatten_string
    driver(flatten_string.main, sys.argv)

Because flatten does not have to wait for anything, it returns
a Deferred that has already fired. The d.addCallback sees this
and calls the callback immediately. Client._continue calls the
next iteration of the test by calling self.request again, and
the stack blows up. This is perfectly reasonable and standard
behaviour for Deferreds, so I should be doing the iteration in
some other way, probably not using Client at all. What I was
hoping for was a pattern for how to transform the code to avoid
the problem; I suspect the answer is to use iteration instead
of recursion. It might even be that none of benchlib.py is
usable directly.

Or maybe putting the flatten() calls into a thread would work?
But that runs the risk of race conditions, if it finishes
before the callback is added.

Peter.