[Twisted-web] Streaming render

glyph at divmod.com glyph at divmod.com
Thu Sep 6 21:43:19 EDT 2007

On 6 Sep, 11:59 pm, mithrandi-twisted-web at mithrandi.za.net wrote:
>I'd like to start streaming the document as soon as I start receiving
>the results of the operation, but I'm stumped on how to actually
>implement this. If I return a deferred from the render method, I cannot
>fire this deferred until all of the rows have been built up.

Use a recursive generator that yields Deferreds for loading each row.

Here's an example - which I even looked at in a web browser :) - that 
should help to get you started.  (The athena stuff is beside the point, 
it's just the quickest way I know to get a server serving a Nevow 

Warning: I haven't tested this on really large datasets, so it might 
have disastrous performance consequences.  I don't *think* that it will, 
but I know there are some booby-traps in the rendering pipeline to snare 
the unwary.

# -- cut here for slow.py --
# Run quickly: twistd -n athena-widget --element slow.SlowElement

from twisted.internet import defer
from twisted.internet import reactor
from nevow.loaders import stan
from nevow.tags import div, directive
from nevow.athena import LiveElement
from nevow.page import renderer

def deferLater(n, result=None):
    Why isn't this in Twisted yet?  Is it?
    d = defer.Deferred()
    reactor.callLater(n, d.callback, result)
    return d

def delayedResults(results):
    Yield results.... slowly.....
    yield div[results.pop(0)]
    if results:
        def more(result):
            return delayedResults(results)
        yield deferLater(2.0).addCallback(more)

class SlowElement(LiveElement):
    Render a list of 1 to 10 really slowly.
    def start(self, ctx, data):
        Give some data, wait for next result...
        return delayedResults(map(str,range(10)))
    docFactory = stan(div(render=directive("start")))

More information about the Twisted-web mailing list