[Twisted-web] some questions about twisted.web

[glyph at divmod.com]

A lot's already gone on in this thread, but I don't think the original 
questions were answered too well.  (Apologies if I've missed a message 
which did so, there were a lot of them.)

Since you asked, I'll leave the filing of the tickets to you :).

I appreciate the thorough analysis of the issues that one encounters 
when trying to write a customized web server using Twisted.  If you do 
go ahead and file these tickets and attach a keyword to them ("tape"?), 
I'll make sure that somebody has a look at them during the next Twisted 
sprint in Cambridge.

Speaking of which, we should set up another one of those.

On 7 Apr, 03:46 pm, jack at chesspark.com wrote:
>I just wrote a simple tool to make my AJAX app development cycle
>better.  It's a tiny web server that serves up the current directory
>along side some reverse proxies.  You can find the current code here:

Cool.
>http://github.com/metajack/tape/tree/master

It seems like the need for this tool could have been obviated if "twistd 
web" had better support for composing plugins of its own.  I'd encourage 
you to look at 'twistd web --help' and see if maybe some future release 
can provide all the functionality you want.

For example, we've got "twistd web --path" and "twistd web --wsgi", but 
no convenient way to combine them to say "My WSGI app is over here, 
mapped to this URL, but it's got some static content over there, mapped 
to that URL".

If what I'm describing makes sense to you, please go ahead and file an 
enhancement ticket to add this feature.  I'm tempted to dive right in 
and start describing some implementation ideas, but knowing me this 
email's going to be long enough already.
>* HTTPFactory seems only able to log to files.  It takes a path name,
>not a file like object.  This makes it very difficult to log to
>stdout, as I have to override log() in my own class.  This in turns
>means cherry picking a global out of twisted.web - fun!

This is at _least_ three bugs.

HTTPFactory should take a FilePath-like object (not a file-like object, 
since it may need to control when the log is opened and closed.

_openLogFile is specifically documented as being for overriding in 
subclasses, yet it's private.  That doesn't make much sense.  Luckily 
the interface is broken anyway, so when we switch over to using a 
FilePath we can switch.

While we're at it, twisted.python.logfile should really be updated to 
use FilePath as well, rather than direct os.path operations.

The CLF-formatting logic really ought to be separated from the code that 
decides to actually write it to a file.  Off the top of my head it 
really seems like this should be a method on Request, but a free 
function would be fine too.

In addition, there's this bug:

    http://twistedmatrix.com/trac/ticket/75

which is maybe too vague to actually be actionable, but I think the idea 
was to publish Request objects with a custom format string via 
twisted.python.log.
>* log() never gets called when a ReverseProxyResource is used.  It
>appears as though request.finish() is never called, though I wasn't
>able to fully track this down.  I find this behavior very odd.

This one's an existing bug:

    http://twistedmatrix.com/trac/ticket/2677

Please feel free to attach commentary.
>* static.File, when given a directory, creates instances of itself to
>handle children.  This code completely fails for me with when I
>subclassed static.File.  Note that my class only has a few
>construction parameters, while static.File has 5.  This is not
>documented.  It would be nice if the code could detect this case and
>tell me that I have to overload createSimilarFile.

Again multiple tickets:

  * In the long term, static.File really ought to compose in a FilePath, 
not inherit from it.  I don't know if it's possible to salvage 
static.File in its current state; I think we need a new static-content 
class that generally works the right way.

For what it's worth, in your use case, you may want to override 
createSimilarFile().  In the meanwhile, before we have a new-and- 
improved File resource,

  * The parameters to __init__ should obviously be documented.

  * createSimilarFile should be documented.
>* The only way to stick something in the tree at an arbitrary location
>seems to be to walk the tree to that spot, creating dummy locations as
>you go.  I find it extremely weird coming from several other web
>frameworks.  There is nothing else but the concept of "hey you!  get
>me child X". This makes it pretty hard to implement anything better
>than walking the tree since state would have to be collected over the
>traversal.

In Nevow and web2, locateChild allows you to consume multiple segments 
at once.  The plan has always been to port these to twisted.web, but I 
can't find a ticket for that plan.

We've discovered some problematic issues in the specifics of the way 
locateChild works; when we design a new one for twisted.web it will 
probably work subtly differently.  But we do need a ticket to discuss 
that work.  I am embarrassed to notice that none of this has been made 
public yet.

So, there's a ticket for general resource lookup improvement.  But I 
think independent of that there's also a need here for an "overlay" 
resource, which maps specific URIs to different resources, as if they 
all lived at the root.  I'm going to handwave here a bit (don't think 
too hard about the fact that the first argument below is just a string, 
or how exactly dispatch works), but for example, let's say you've got a 
site which has been through a couple of transitions and you want to keep 
all of your URLs working.

currentBlog = AwesomeTwistedBlog()
o = Overlay(default=currentBlog)
bloggerCompat = BloggerCompatibilityResource(currentBlog)
o.mapPath("/2008/*", bloggerCompat)
o.mapPath("/2007/*", bloggerCompat)
o.mapPath("*.php", WordpressCompatibilityResource(currentBlog))
o.mapPath("/static", FilePath("some-data"))

Oops, when I talked about composing plugins above I said I wasn't going 
to talk about implementation details.  I guess I lied.  This is what I 
was talking about; "Overlay()" is the resource which we'd need to 
implement 'twistd web --wsgi my.application.here --at /apps/my-app 
--path ./app/my-static-content --at /static'.

So again, if this sounds useful, go ahead and file a ticket.  Although 
the one for the command-line stuff should be separate from this 
substrate, they should link to each other.
>* Related to the above, the handling of foo vs. foo/ is pretty
>confusing.  foo/ is considered the '' child of foo.  This is pretty
>yuck to me.  Unfortunately, I don't have any creative suggestions
>about how to do it better right now.

The web sucks.  There actually *is* a pretty significant semantic 
distinction between foo and foo/ in a URI.  In particular, in order to 
generate relative links properly you _really_ need to care about that 
distinction.

Even the thing that dotz mentioned, addSlash, is a bit of a misfeature; 
it's good convenience functionality but the interface to it is magical 
and unclear.

This is the one thing I think isn't really a valid bug unless you have a 
better idea :).
>* ReverseProxyResource returns nothing if you don't give it a '/' at
>the end.  I had to work around this by doing a redirect in the
>subclass and then returning a new ReverseProxyResource when the ''
>child is accessed.  It seems that if ReverseProxyResource's path is
>'', it just keels over with no error.

This could do with a bit more detail (mostly about what "keels over" 
means), but certainly sounds like a valid bug.