[Twisted-web] Streaming render

[glyph at divmod.com]

On 03:01 pm, mithrandi-twisted-web at mithrandi.za.net wrote:
>* glyph at divmod.com <glyph at divmod.com> [2007-09-07 01:43:19 -0000]:
>>Use a recursive generator that yields Deferreds for loading each row.
>
>Thanks for the example! Combined with a DeferredQueue to pass the data
>from the callback to the rendering "loop", this approach seems to work
>perfectly.

No problem!  Your Fan Club Dollars At Work!22.
>>Here's an example - which I even looked at in a web browser :) - that
>>should help to get you started.  (The athena stuff is beside the 
>>point,
>>it's just the quickest way I know to get a server serving a Nevow 
>>page...)
>
>The athena-widget plugin is definitely very handy for quick examples
>like this.

We've also been discussing a slightly more general "twistd nevow" 
plugin.  More on this as it develops.
>>Warning: I haven't tested this on really large datasets, so it might 
>>have
>>disastrous performance consequences.  I don't *think* that it will, 
>>but I
>>know there are some booby-traps in the rendering pipeline to snare the
>>unwary.
>
>As far as I can tell, nothing particularly nasty is happening when I 
>use
>this; if the overhead of producing each row is very low, all the
>indirection through generators / deferreds might be costly, but that's
>certainly not the case for my code (the database and formatting
>operations are relatively expensive).

Good to know.
>http://twistedmatrix.com/trac/ticket/1875
>
>I'm not really sure what I can do to drive the ticket along, but if
>there is something, please let me know.

I vaguely remembered this ticket when I wrote that comment :-).  Some of 
the things that are blocking it are on my list of stuff to get back to 
when I get back to Twisted development at all.