[Twisted-web] html cache with timeout
Matt Goodall
matt at pollenation.net
Tue Feb 1 10:15:01 MST 2005
On Tue, 2005-02-01 at 16:05 +0000, Valentino Volonghi aka Dialtone
wrote:
> On Tue, 01 Feb 2005 12:58:49 GMT, Valentino Volonghi aka Dialtone <dialtone at divmod.com> wrote:
>
> The first version of the patch didn't actually work. But I wrote a new version, also thanks to fzZzy and this time it works although it has a flaw since in weever caching the content slot (which is filled with a Fragment) results in 2 big red Nones and the rendered fragment.
>
> As I said in the first mail you can use caching with:
>
> t.cached(name=some_sensible_name, lifetime=MAX_LIFE)[cached_content]
>
> This patch provides, probably, the finest granularity in caching the rendering.
>
> Anyway the patch is below:
I think it would be much better if the _CACHE module-scope dict was
replaced with an object remembered in the context. There are a couple of
reasons for this:
* We can have persistence to the file system when necessary.
* We can remember a cache manager on a resource to allow drop-in
components (and their child resources) to manage their own
caching. This also allows some root resource class to have
multiple instances, where the interface names used as the cache
keys will likely be the same, to be deployed under a single
site.
* The cache manager API can be extended in the future to allow
manual clearing of cache items, i.e. some public web UI can
cache parts of the page indefintely and an admin UI (that shares
the same cache manager) can clear cached data as objects are
modified.
There are also a couple of features that I can see stan.cached "growing"
later on. I've mentioned some of these on IRC.
* Cache scope, i.e. application vs session. As I've said on IRC, I
can see a real use case for session-scoped caching, i.e. I get
my cached version; you get yours.
* It might be nice to allow timeouts to be defined as "every
hour", "every fifteen minutes", "at 12am". Yeah, I'm talking
cron-like ;-).
Hmm, one last idea is cache groups. I think this is especially
applicable to the above idea of having an API to clear cache objects.
Say some part of the page includes content from two objects: a Foo with
id 3 and a Bar with id 8. The fragment could be cached against the key
((Foo,3),(Bar,8)). If some user then changed the Foo,3 object it would
clear (Foo,3) cached objects; if Bar,8 was changed it would clear
(Bar,8) cached objects. Either one would remove the ((Foo,3),(Bar,8))
cached content.
Obviously, it would be up to the application to choose its keys carefull
but, basically, if the equivalent of "(Foo,3) in ((Foo,3),(Bar,8))"
succeeds then the object would be cleared.
I don't think we have to add all these features right now as long as the
initial API takes these sorts of use cases into consideration.
Cheers, Matt
>
> Index: nevow/tags.py
> ===================================================================
> --- nevow/tags.py (revision 1136)
> +++ nevow/tags.py (working copy)
> @@ -25,7 +25,7 @@
> """
>
>
> -from nevow.stan import Proto, Tag, directive, raw, xml, CommentProto, invisible, slot, cdata
> +from nevow.stan import Proto, Tag, directive, raw, xml, CommentProto, invisible, slot, cdata, cached
>
>
> comment = CommentProto()
> @@ -62,7 +62,9 @@
> def inlineJS(s):
> return script(type="text/javascript", language="JavaScript")[xml('\n//<![CDATA[\n%s\n//]]>\n' % s)]
>
> -__all__ = tags + ['invisible', 'comment', '_dir', '_del', '_object', '_map', 'drange', 'Tag', 'directive', 'xml', 'raw', 'slot', 'cdata', 'inlineJS'] + ['_%s' % x for x in range(100)]
> +__all__ = tags + ['invisible', 'comment', '_dir', '_del', '_object',
> + '_map', 'drange', 'Tag', 'directive', 'xml', 'raw',
> + 'slot', 'cached', 'cdata', 'inlineJS'] + ['_%s' % x for x in range(100)]
>
>
> ########################
> Index: nevow/flat/flatstan.py
> ===================================================================
> --- nevow/flat/flatstan.py (revision 1136)
> +++ nevow/flat/flatstan.py (working copy)
> @@ -9,7 +9,7 @@
> from nevow import util
> from nevow.stan import Proto, Tag, xml, directive, Unset, invisible
> from nevow.inevow import IRenderer, IRendererFactory, IGettable, IData
> -from nevow.flat import precompile, serialize
> +from nevow.flat import precompile, serialize, iterflatten
> from nevow.accessors import convertToData
> from nevow.context import WovenContext
>
> @@ -226,6 +226,56 @@
> return serialize(original.default, context)
> return serialize(data, context)
>
> +_CACHE = {}
> +from time import time as now
> +from cStringIO import StringIO
> +from twisted.internet import defer
> +def CachedSerializer(original, context):
> + cached = _CACHE.get(original.name, None)
> + life = now()-original.lifetime
> + if cached and cached[0] > life:
> +## print "="*20
> +## print cached[0]
> +## print life
> +## print "="*20
> + yield cached[1]
> + return
> +## if cached:
> +## print "="*20
> +## print cached[0]
> +## print life
> +## print "="*20
> + io = StringIO()
> + for child in iterflatten(original.children, context, io.write,
> + lambda item: True):
> + if isinstance(child, tuple):
> + childDeferred, childReturner = child
> +
> + d = defer.Deferred() ## A new deferred for the outer loop, whose result
> + ## we don't care about, because we don't want the outer loop to write
> + ## anything when this deferred fires -- only when the entire for loop
> + ## has completed and we have all the "children" flattened
> +
> + def innerDeferredResultAvailable(result):
> + childReturner(result) ## Cause the inner iterflatten to continue
> + d.callback('') ## Cause the outer iterflatten to continue
> + return ''
> +
> + childDeferred.addCallback(innerDeferredResultAvailable)
> +
> + ## Make the outer loop wait on our new deferred.
> + ## We call the new deferred back with ''
> + ## Which will cause the outer loop to write '' to the request,
> + ## which doesn't matter. It will then call our "returner",
> + ## which is just the noop lambda below, because we don't care
> + ## about the return result of the new deferred, which is just
> + ## ''
> +
> + yield d, lambda result: None
> + result = io.getvalue()
> + _CACHE[original.name] = (now(), result)
> + yield result
> +
> def ContextSerializer(original, context):
> originalContext = original.clone(deep=False)
> originalContext.precompile = context and context.precompile or False
> Index: nevow/__init__.py
> ===================================================================
> --- nevow/__init__.py (revision 1136)
> +++ nevow/__init__.py (working copy)
> @@ -182,6 +182,7 @@
> nevow.flat.flatstan.RendererSerializer nevow.inevow.IRenderer
> nevow.flat.flatstan.DirectiveSerializer nevow.stan.directive
> nevow.flat.flatstan.SlotSerializer nevow.stan.slot
> +nevow.flat.flatstan.CachedSerializer nevow.stan.cached
> nevow.flat.flatstan.ContextSerializer nevow.context.WovenContext
> nevow.flat.flatstan.DeferredSerializer twisted.internet.defer.Deferred
> nevow.flat.flatstan.DeferredSerializer twisted.internet.defer.DeferredList
> Index: nevow/stan.py
> ===================================================================
> --- nevow/stan.py (revision 1136)
> +++ nevow/stan.py (working copy)
> @@ -119,8 +119,33 @@
> """
> raise NotImplementedError, "Stan slot instances are not iterable."
>
> +class cached(object):
> + """Marker for cached content
> + """
> + __slots__ = ['name', 'children', 'lifetime']
>
> + def __init__(self, name, lifetime=0):
> + self.name = name
> + self.children = []
> + self.lifetime = lifetime
>
> + def __repr__(self):
> + return "cached('%s','%s')" % self.name, self.lifetime
> +
> + def __getitem__(self, children):
> + """cached content is what is being cached
> + """
> + if not isinstance(children, (list, tuple)):
> + children = [children]
> + self.children.extend(children)
> + return self
> +
> + def __iter__(self):
> + """Prevent an infinite loop if someone tries to do
> + for x in cached('foo'):
> + """
> + raise NotImplementedError, "Stan slot instances are not iterable."
> +
> class Tag(object):
> """Tag instances represent XML tags with a tag name, attributes,
> and children. Tag instances can be constructed using the Prototype
>
> _______________________________________________
> Twisted-web mailing list
> Twisted-web at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
More information about the Twisted-web
mailing list