[Twisted-Python] on template processing

[Glyph Lefkowitz]

Hi Kevin!  Sorry for the delay; your message caught me (and I suspect many other people) on vacation.

> On Dec 26, 2015, at 4:08 PM, Kevin Mcintyre <kebin70 at gmail.com> wrote:
> 
> 1) Is template block processing done top to bottom sequentially?  I've included example.py - it's interesting that by having a blocking call first the second request is blocked, but having the blocking call after a non-blocking call the second request is not.  I would've thought it would behave as a deferred list, but looking at _flattenTree and guessing not.  Maybe related to wait_for_it example?

I'm not entirely sure what you mean by "top to bottom" or "sequentially", but I think for all reasonable meanings of these words the answer would be "yes".  However, blocking calls on the main loop always block the entire main loop in Twisted; multiple requests do not interact with each other in twisted.web.template, except for the fact that they share a main loop, just as everything else in Twisted does.

> 2) Is it possible for a Resource to act as an Element too?  I've included a non-working elementresource.py.  I'm a total hack, but I would think that if an instance had a loader attr it could be processable.

A Resource is a Resource; an Element is an Element.  They do different things, and muddling up the request-dispatching API further wouldn't help anything.  Multiple inheritance is always a "you have two problems" sort of scenario.  However, what I think you want (a place where you could conveniently return an Element or a Resource, and the result would be properly adapted) is supported by Klein:

https://github.com/twisted/klein/blob/e336c5b1af25badb3f159ab72b3d1bf392d45ba5/klein/resource.py#L225-L226

You can return an Element from any klein route and it will be turned into a Resource for you.

> 3) Is it possible to include xmlns:t="http://twistedmatrix.com/ns/twisted.web.template/0.1 <http://twistedmatrix.com/ns/twisted.web.template/0.1>" not in a tag itself?  Or perhaps have a tag like 'render-block' that could be transparent-like?

Not built-in, but the https://twistedmatrix.com/documents/15.5.0/api/twisted.web.iweb.ITemplateLoader.html interface allows you to load your template however you like.  I have written a couple of small apps that used html5lib with some post-processing to load templates instead of expat, and if I weren't so lazy I would have contributed one of them to Twisted :).  You could write a loader that wraps the document in an enclosing tag that adds the XML namespace textually, and that might do the trick.  We'd definitely be interested in a contribution that dealt with this.

> 4) Is it possible to have xmlns:t="http://twistedmatrix.com/ns/twisted.web.template/0.1 <http://twistedmatrix.com/ns/twisted.web.template/0.1>" in multiple places in a template?  I have some cases with inline scripts that bonk out with > or < characters.  This is very much related to #3.

"bonk out"?  It sounds like you have a cross-site-scripting attack in the making here.  Messing with how the template gets loaded is not the right way to address it.  Please describe this in more detail so I don't give you advice that is going to get your users pwnt because I don't fully understand your use-case :).

> 5) Is it possible for a render element to return something like "<~sometag t:render>..." and process recursively?

You mean return a literal string, or do you just mean you want an element to return another element from a renderer?

> 6) Is there any examples of connection keep-alive long polling?

Just return NOT_DONE_YET, save the request away, and call request.write/request.finish sometime later.  It's almost too simple to have an example :).

> 7) Examples of request based scoping would be great.  All the examples on http://twistedmatrix.com/documents/13.0.0/web/howto/twisted-templates.html <http://twistedmatrix.com/documents/13.0.0/web/howto/twisted-templates.html> have flatten(None... -  I've included request_scope.py 

Setting arbitrary attributes on the request is a terrible anti-pattern, so you won't find any examples of using the request for that.  Looking at the documentation now, I can see it doesn't really explain how to manage state necessary for rendering an element, and it probably should.  This type of documentation oversight is very common in Twisted, unfortunately, because there's a strange disconnect between the way the average Twisted core developer thinks and the way the average Python developer thinks that continues to mystify me.  In this case, we assume you've put your state on 'self' so the examples all focus on just how to inject it into the template with slots.

This is exemplified by this Stack Overflow question I wrote a while ago - http://stackoverflow.com/questions/7313761/how-do-i-add-two-integers-together-with-twisted - it's an extremely general problem that's hard to pin down.  It shows up particularly often when twisted's users are trying to make multiple protocols, or web resources, interact with each other.  The general question is: how do I use Twisted to manage the state associated with these connections?  What Twisted API should I use to move state between them?  In your case, you're asking how you can use APIs within Element or Resource to move state from your resource's render method to the Element you're instantiating.

The answer is: your Element is just a Python class.  Don't stick stuff onto the request and then pass the request into the Element via the rendering mechanism so that you can pull the state back out again.  Just pass the state that the Element needs to do its job into its constructor.

The reason the docs often don't cover this sort of thing is that it seems bizarre to have to explain how Python classes work; and in fact I'm sure you (and our audience) do know how they work, so an explanation of how to write a constructor seems out of place in the documentation for twisted.web.  I can see that this is a "best practice" that should at least be touched on though, so that people don't encounter this sort of confusion.

> 8) The wait_for_it example, is that meant as a chunked transfer example?  It would be cool to have an example I could open in a browser.  Trying to wrap my head around this and subviews in the meantime.

Chunked transfer is a transport implementation detail; this is meant as an example of how you can use Deferreds.  The problem with demonstrating this concept in a browser is that browsers have weird, arbitrary thresholds for incremental rendering and will generally intentionally avoid partially rendering anything example-sized, instead opting to leave the page blank until the full body has been received. This example will send "before waiting ..." to the browser as soon as the request is sent, but we can't control what the browser actually does with it.  I actually tried to fix this up a few months ago and the rules that browsers use for incremental rendering just completely baffled me; if you can figure out a way to reliably demonstrate this it would be great.

Thanks for using Twisted!

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20160104/25383c8a/attachment.html>