[Twisted-web] Re: Setting up a project.

Govind Salinas blix at sophiasuchtig.com
Wed Aug 6 17:40:49 EDT 2008

On Wed, Aug 6, 2008 at 11:19 AM, David Bolen <db3l.net at gmail.com> wrote:
> "Govind Salinas" <blix at sophiasuchtig.com> writes:
 -- snip
>> Let me give you some information about the set up I have .  Basically
>> I have a python program running somewhere.  I will have a default set
>> of files that serve the content a particular way.  The default set
>> will contain mako templates[1], style sheets and perhaps some images.
>> So I need a way to kick off the mako templates based on the URL.  The
>> templates may include each other.
>> Here is the layout
>> content-dir/
>>    __config__.py -- This is a file that currently just sets up the mapping
>>                             between a regex that selects a page and a path to
>>                             where that page's template is under the content-dir
>>    main.html -- The html files here are really the mako templates
>>    main.css
>>    etc...
>> My server class takes in a template dir, loads the config and starts a
>> server.  When it gets a request it matches it against the regexes and
>> runs the template.  There should be some initialization that sets up
>> objects that get passed to the template (in addition to values from
>> the regex parsing). This is all pretty brain-dead code.
> Are you saying "brain-dead" in a negative way here, or to imply that
> it's boilerplate or something you'd prefer the framework do for you?
> With twisted.web, you're pretty much in charge of this sort of stuff.

I was referring to the code I was writing.  I guess I would consider it
boiler plate.  I think some more complete examples in the howto
would have helped me here.  I seemed to get the gist of what
resources were supposed to do, but wasn't sure how to put
it all together.

> I think you've pretty much hit most of the items you need to cover.
> For URL traversal, it's not clear from your description if you are
> using the regex mapping to also find the Resource object or just for
> template lookups by the Resource objects.  Either approach is fine.
> Twisted.web itself will traverse a URL by following the chain of
> Resource objects and their children.  This can be much more dynamic
> than a fixed URL->object mapping definition since the Resources can be
> created on the fly when needed.  If your Resource tree is static, then
> a fixed mapping is fine too.

I had tried using Django for what I was doing but I did not sure if it
would end up as a nice fit.  When I started using twisted, I borrowed
the regex->page mapping from them.  This is how I implemented it:

    def _parse_setup(self, path):
        self._config = {}
        execfile(path, {}, self._config)
        if 'url_map' not in self._config:
            return False
        self._mappings = []
        for exp, f in self._config['url_map']:
            f = f.split('/')
            f.insert(0, self._template_dir)
            f = os.path.join(*f)
            self._mappings.append((re.compile(exp), f))
        return True

    def render_GET(self, request):
        path = request.path[1:]
        uri = request.uri[1:]
        for mapping, f in self._mappings:
            match = mapping.match(uri)
            if match:
                vars = match.groupdict()
                print 'serving:', f
                lookup = TemplateLookup(directories=['/'])
                return Template(filename=f, lookup=lookup).render(**vars)
        path = os.path.join(self._template_dir, path)
        if os.path.isfile(path):
            f = None
                idx = path.rfind('.')
                print 'serving static:', path, path[idx + 1:]
                type = _mime_types.get(path[idx + 1:], 'text/plain')
                request.setHeader('content-type', type)
                print request.headers, type
                f = file(path, 'r')
                return f.read()
                print 'failed to serve:', path
                if f:
            print '%s does not exist' % path
        return self._page_not_found

I was not too clear on how the resources would be used, so I chose
this.  And the files/paths are all static, even if the content is not.
One of the things I want to be able to do is allow the user to have
a choice between templates or be able to write their own.  The url
mapping seemed like a good way of soft coding that so that a
different template could organize their code differently.

> Generally what I've done in the past is build up the fixed portion of
> my Resource tree during server initialization.  My Resource classes
> have internal definitions for their template name (as a relative path
> when using directories of templates) and are just handed the template
> loader during construction, which they later use for rendering.  The
> template loader is given the template root directory at
> initialization.  The trade-off with your regex mapping is that you
> have to keep two fairly separate bits of code in sync, where I have
> the template information inside the object definition, but in my case
> it can also be more work to get a global view of template assignments.
> In terms of your question on static files (css/images/js), I just
> place those into their own sub-tree and then defined a static.File()
> object to serve them.  This also has an advantage that if you place
> your server behind, say, Apache, in production use, you can define
> rules on the Apache side to let it directly serve the static files and
> never involve your code, for some improved efficiency.

That sounds like a good idea, I will do that.

> Another area that you may need to implement your own support would be
> for handling authorization (twisted.web provides support for managing
> sessions via cookies, but you have to decide when to render an
> authentication page and then store any needed authentication
> information in the session).

I have no use for that yet.  However I will eventually need to do http
and/or ssh authentication (the ability to POST updates to the
repository is a planned feature).  So this will come in handy.

>> I was hoping for some guidance on a better strategy for doing this.
>> Any help would be appreciated.  Bonus points if it is something that
>> could easily be re-used on another webserver if someone wanted to host
>> this on an existing website.
> I think you need to help defined what parts of the above you perceive
> as needing to be "better".  I've implemented several servers (many of
> which were embedded in a larger application) using the above approach
> and it seems just fine to me.  As to re-usability, that's pretty much
> the same question for any code you write - e.g., if you're doing
> authentication support, making it reusable is pretty much the same as
> making any other piece of code reusable.
> If you're looking for a higher level framework approach (much more
> functionality "for free"), then twisted.web itself may not be the
> ideal solution for you.  In the twisted world, nevow may provide a bit
> more of what you're looking for, and of course there are other high
> profile high-level web frameworks like django, turbogears, pylons, etc...
I took a quick look at nevow and I think I like the mako templates
better.  It behaves more that a "python server pages" type of
template, which is conceptually similar to the little web programming
that I have done in the past.

> As a more concrete example, here are some snippets of one of my recent
> web server modules (in commercial use for the past year or so).  Note
> this is a straight cut 'n paste, so there's various logic in addition
> to the pure web processing, and probably references to functions/objects
> that won't fully be explained.

-- snip

> Hope there's not too much "noise" in the code to prevent it from being
> helpful.

Actually that was perfect, I have a much better understanding of how things
are supposed to work now.  I appreciate the help.


More information about the Twisted-web mailing list