[Twisted-web] some questions about twisted.web
glyph at divmod.com
glyph at divmod.com
Tue Apr 7 22:57:16 EDT 2009
A lot's already gone on in this thread, but I don't think the original
questions were answered too well. (Apologies if I've missed a message
which did so, there were a lot of them.)
Since you asked, I'll leave the filing of the tickets to you :).
I appreciate the thorough analysis of the issues that one encounters
when trying to write a customized web server using Twisted. If you do
go ahead and file these tickets and attach a keyword to them ("tape"?),
I'll make sure that somebody has a look at them during the next Twisted
sprint in Cambridge.
Speaking of which, we should set up another one of those.
On 7 Apr, 03:46 pm, jack at chesspark.com wrote:
>I just wrote a simple tool to make my AJAX app development cycle
>better. It's a tiny web server that serves up the current directory
>along side some reverse proxies. You can find the current code here:
It seems like the need for this tool could have been obviated if "twistd
web" had better support for composing plugins of its own. I'd encourage
you to look at 'twistd web --help' and see if maybe some future release
can provide all the functionality you want.
For example, we've got "twistd web --path" and "twistd web --wsgi", but
no convenient way to combine them to say "My WSGI app is over here,
mapped to this URL, but it's got some static content over there, mapped
to that URL".
If what I'm describing makes sense to you, please go ahead and file an
enhancement ticket to add this feature. I'm tempted to dive right in
and start describing some implementation ideas, but knowing me this
email's going to be long enough already.
>* HTTPFactory seems only able to log to files. It takes a path name,
>not a file like object. This makes it very difficult to log to
>stdout, as I have to override log() in my own class. This in turns
>means cherry picking a global out of twisted.web - fun!
This is at _least_ three bugs.
HTTPFactory should take a FilePath-like object (not a file-like object,
since it may need to control when the log is opened and closed.
_openLogFile is specifically documented as being for overriding in
subclasses, yet it's private. That doesn't make much sense. Luckily
the interface is broken anyway, so when we switch over to using a
FilePath we can switch.
While we're at it, twisted.python.logfile should really be updated to
use FilePath as well, rather than direct os.path operations.
The CLF-formatting logic really ought to be separated from the code that
decides to actually write it to a file. Off the top of my head it
really seems like this should be a method on Request, but a free
function would be fine too.
In addition, there's this bug:
which is maybe too vague to actually be actionable, but I think the idea
was to publish Request objects with a custom format string via
>* log() never gets called when a ReverseProxyResource is used. It
>appears as though request.finish() is never called, though I wasn't
>able to fully track this down. I find this behavior very odd.
This one's an existing bug:
Please feel free to attach commentary.
>* static.File, when given a directory, creates instances of itself to
>handle children. This code completely fails for me with when I
>subclassed static.File. Note that my class only has a few
>construction parameters, while static.File has 5. This is not
>documented. It would be nice if the code could detect this case and
>tell me that I have to overload createSimilarFile.
Again multiple tickets:
* In the long term, static.File really ought to compose in a FilePath,
not inherit from it. I don't know if it's possible to salvage
static.File in its current state; I think we need a new static-content
class that generally works the right way.
For what it's worth, in your use case, you may want to override
createSimilarFile(). In the meanwhile, before we have a new-and-
improved File resource,
* The parameters to __init__ should obviously be documented.
* createSimilarFile should be documented.
>* The only way to stick something in the tree at an arbitrary location
>seems to be to walk the tree to that spot, creating dummy locations as
>you go. I find it extremely weird coming from several other web
>frameworks. There is nothing else but the concept of "hey you! get
>me child X". This makes it pretty hard to implement anything better
>than walking the tree since state would have to be collected over the
In Nevow and web2, locateChild allows you to consume multiple segments
at once. The plan has always been to port these to twisted.web, but I
can't find a ticket for that plan.
We've discovered some problematic issues in the specifics of the way
locateChild works; when we design a new one for twisted.web it will
probably work subtly differently. But we do need a ticket to discuss
that work. I am embarrassed to notice that none of this has been made
So, there's a ticket for general resource lookup improvement. But I
think independent of that there's also a need here for an "overlay"
resource, which maps specific URIs to different resources, as if they
all lived at the root. I'm going to handwave here a bit (don't think
too hard about the fact that the first argument below is just a string,
or how exactly dispatch works), but for example, let's say you've got a
site which has been through a couple of transitions and you want to keep
all of your URLs working.
currentBlog = AwesomeTwistedBlog()
o = Overlay(default=currentBlog)
bloggerCompat = BloggerCompatibilityResource(currentBlog)
Oops, when I talked about composing plugins above I said I wasn't going
to talk about implementation details. I guess I lied. This is what I
was talking about; "Overlay()" is the resource which we'd need to
implement 'twistd web --wsgi my.application.here --at /apps/my-app
--path ./app/my-static-content --at /static'.
So again, if this sounds useful, go ahead and file a ticket. Although
the one for the command-line stuff should be separate from this
substrate, they should link to each other.
>* Related to the above, the handling of foo vs. foo/ is pretty
>confusing. foo/ is considered the '' child of foo. This is pretty
>yuck to me. Unfortunately, I don't have any creative suggestions
>about how to do it better right now.
The web sucks. There actually *is* a pretty significant semantic
distinction between foo and foo/ in a URI. In particular, in order to
generate relative links properly you _really_ need to care about that
Even the thing that dotz mentioned, addSlash, is a bit of a misfeature;
it's good convenience functionality but the interface to it is magical
This is the one thing I think isn't really a valid bug unless you have a
better idea :).
>* ReverseProxyResource returns nothing if you don't give it a '/' at
>the end. I had to work around this by doing a redirect in the
>subclass and then returning a new ReverseProxyResource when the ''
>child is accessed. It seems that if ReverseProxyResource's path is
>'', it just keels over with no error.
This could do with a bit more detail (mostly about what "keels over"
means), but certainly sounds like a valid bug.
More information about the Twisted-web