[Twisted-Python] breaking distrib.ResourceSubscription

Glyph Lefkowitz glyph at twistedmatrix.com
Tue Sep 24 20:28:33 EDT 2002

On 24 Sep 2002 15:11:41 -0700, Kevin Turner <acapnotic at twistedmatrix.com> wrote:
> On Tue, 2002-09-24 at 06:57, Glyph Lefkowitz wrote:
> [...]
> > ResourcePublisher instance has no attribute 'remote_wasModifiedSince'
> I'm surprised that this is in the logs much, since it should only happen
> for distributed servers that set a Last-Modified header.  Are personal
> servers serving that many static files?  I would have thought that would
> be handled by ~/public_html.

The tracebacks I was seeing were for wikis and viewcvs; either moin is setting
those headers or it's serving the images / static data from a personal server

> > I thought that I'd mentioned this over IRC before, but let me stress it: I
> > think that twisted.web.server is implementing *WAAAAY* too much of HTTP right
> > now. 
> Interesting position.  I know you're not a fan of complete support for a
> protocol when you believe the protocol itself is broken, and I'll admit
> that nobody actually uses ETags, but I would hardly say that
> Last-Modified and HEAD are obscure or unnecessary.

Nor would I.  After much discussion with itamar I conceded that HEAD ought to
cause the body of the request to be eaten there before being sent to the

What I mean here is that the actual module, twisted.web.server, is being too
smart about stuff like Last-Modified and ETags.  HTTP should be fully
implemented, but parts of the responsibility lie elsewhere.
twisted.web.Resource subclasses can implement those themselves if they want to,
because the 99.9% cases should be handled: static.File will handle filesystem
content and woven.* will handle dynamic content.  We can also provide a helper
for mix-in that does conditional-rendering stuff.

> > First of all, this is killing performance -- you have a minimum of 4 or 5
> > PB server/client interactions now for each conditional request instead of 2.
> > Those method calls are not cheap.
> Recognized.  Although I thought I only increased the count by one, not
> two or three?

Looking at it now, I guess ETags and last-modified support are exclusive.  (Is
this correct, protocol-wise?)  However, each call that requires an answer is
minimum of 2 interactions (request/response).  Plus most resources will want to
do a .write().

> Also take into consideration that these calls only happen for the
> *conditional* requests.

The only heuristic I have for how often these happen are the once-per-minute
tracebacks I was seeing while bots were crawling the website yesterday.  So it
looked quite often. :)

> > Second, this is exploding the complexity of the webserver: its goal should
> > be to get to the endpoint resource *as simply as possible* and then invoke
> > endpoint behavior on that resource.  If clever header-parsing things need
> > to happen, then we need to make a lower-level 'request' method and make
> > *it* clever, not do request processing before the request is dispatched.

> This I agree with.  I knew something was wrong when I had three remote
> methods in ResourcePublisher all independently doing getResourceFor.  It
> would be more symmetric if Request.process was invoked on the distri --

Right. :-)

> ggrngh.  What is Request.process anyway.  Is it just supposed to do
> site.getResourceFor and pass itself off?  Yes.  All the rest of the junk
> in there is stuff I added (though the HEAD/supportedMethods stuff was
> quite some time ago) with the possible exception of the
> util.Unauthorized block.  And it's probably all logic that belongs in
> web..Resource rather than Request.

HEAD definitely ought to still have a little bit of logic in Request, since it
should discard any body that the resource attempts to return.  However, the
rest of what you're saying is definitely...

> ([...] just re-iterating what you've been telling me.)

> So how do we fix it?  You're suggesting that Request.process call
> Resource.request, which contains most of the stuff currently in
> Request.process, and Resource.request calls Resource.render if it's not
> a conditional request?

See above for my suggestion.  I think that Resource is too low-level for this
sort of thing (since the default implementation of wasModifiedSince is going to
be 'return 1', why invoke it at all by default?).  I don't feel strongly that
my suggestion is better than this one though, if you think that dynamic
resource implementors are going to want to use this functionality a lot.
(Thoughts, Donovan?)

> Sounds like we need an inter-version test harness.  Runs peer1 under
> version X and peer2 on Y, or starts server on version X, pickles it, and
> re-starts under Y, etc.

Yes.  This is something that I think radix hacked up manually a little while
back, but it needs to be added to accepttests.

Sounds like we're in general agreement, then...

 |    <`'>    |  Glyph Lefkowitz: Traveling Sorcerer   |
 |   < _/ >   |  Lead Developer,  the Twisted project  |
 |  < ___/ >  |      http://www.twistedmatrix.com      |
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://twistedmatrix.com/pipermail/twisted-python/attachments/20020924/879e7dc8/attachment.pgp 

More information about the Twisted-Python mailing list