[Twisted-web] mailman with twisted

Andrea Arcangeli andrea at cpushare.com
Thu Feb 10 15:14:45 MST 2005


On Thu, Feb 10, 2005 at 10:35:42PM +0200, Tommi Virtanen wrote:
> Pass in a different mode.
> 
> def listenUNIX(self, address, factory, backlog=5, mode=0666, wantPID=0):
> 
> UNIXServer should accept the same args.

Ok fine, the mode should be enough because I'm lucky and the directory
has g+s.

> If a file by given name exists, bind(2) returns -1, EADDRINUSE.
> Twisted is just passing that straight-through to you.

29253 bind(5, {sa_family=AF_FILE,
path="/var/lib/mailman/.twistd-web-pb"}, 33) = -1 EADDRINUSE (Address
already in use)

Ok my bad ;) it was a kernel issue here.

		err = path_lookup(sunaddr->sun_path, LOOKUP_PARENT, &nd);
		if (err)
			goto out_mknod_parent;
[..]
out_mknod_parent:
	if (err==-EEXIST)
		err=-EADDRINUSE;

I didn't think the check should have been made on the existence of the
dentry, but only on the fact that some other task was actively listening
to such socket already.

It would be easy to relax this restrictive API (perhaps with a check
that the process uid is the same as the socket uid) and then twisted
would just work fine even if it doesn't delete the file but it would
probably break the standard API.

I can workaround it outside twisted easily (an fuser + rm as root before
starting the app, is sure enough ;)

So at least those two problems are clear thanks.

> > At least there should be an option to auto-delete it.
> 
> Maybe, but that option would need to default to False.

Agreed.

> I consider to be too risky to automate in a library. If another process
> is listening on that socket, removing it just hides that fact.

Correct.

> Of course, if you really want it, you could just run
> 
> try:
>     os.unlink(sockName)
> except OSError, e:
>     if e.errno == errno.ENOENT:
>         pass
>     else:
>         raise
> 
> before binding.

The equivalent of fuser before OSError would be best, fuser in this case
is the userspace equivalent of checking the unix_socket_table on the
kernel side. (except it's racy if done in userspace, but good enough to
be used to avoid the basic mistakes of deleting a socket in-use).
A connect would do the trick too (again racy but good enough), if it
returns ECONNREFUSED it means we can delete it.

About the other problem of nevow+distrib, it seems you found the right
issue on IRC, some bugfix has been forgotten here:

http://divmod.org/users/roundup.twistd/nevow/file24/nevow-distrib-2.diff
http://divmod.org/users/roundup.twistd/nevow/file25/nevow-flatstan.diff

Anyway I'm following your suggestion to use eoc instead of mailman, that
seems to be the quickest to setup and it doesn't require web
subscription, but certainly it would be still nice to know the solution
of nevow+mailman just as an exercise ;). I guess reverse proxying as you
suggested on irc would have worked much better than distrib ;)

there are apparently 5 pending things missing in nevow trunk:

1) the above distrib fix
2) the static.File fix posted here for twisted that affects nevow too
3) dialtone's tags.cached
4) dialtone's rend.Page cache improved with cacheTimeout logic.
5) minor non interesting runtime fix in vhost.py

The latter 4 entries are appened here (I hope I didn't add typos while
merging the static.File fix by hand, the mailer got it screwed in the
original submission from Mike Marchionna).

Thanks for the help!

Index: Nevow/nevow/tags.py
===================================================================
--- Nevow/nevow/tags.py	(revision 1192)
+++ Nevow/nevow/tags.py	(working copy)
@@ -25,7 +25,7 @@
 """
 
 
-from nevow.stan import Proto, Tag, directive, raw, xml, CommentProto, invisible, slot, cdata
+from nevow.stan import Proto, Tag, directive, raw, xml, CommentProto, invisible, slot, cdata, cached
 
 
 comment = CommentProto()
@@ -62,7 +62,9 @@
 def inlineJS(s):
     return script(type="text/javascript", language="JavaScript")[xml('\n//<![CDATA[\n%s\n//]]>\n' % s)]
 
-__all__ = tags + ['invisible', 'comment', '_dir', '_del', '_object', '_map', 'drange', 'Tag', 'directive', 'xml', 'raw', 'slot', 'cdata', 'inlineJS'] + ['_%s' % x for x in range(100)]
+__all__ = tags + ['invisible', 'comment', '_dir', '_del', '_object',
+                  '_map', 'drange', 'Tag', 'directive', 'xml', 'raw',
+                  'slot', 'cached', 'cdata', 'inlineJS'] + ['_%s' % x for x in range(100)]
 
 
 ########################
Index: Nevow/nevow/__init__.py
===================================================================
--- Nevow/nevow/__init__.py	(revision 1192)
+++ Nevow/nevow/__init__.py	(working copy)
@@ -183,6 +183,7 @@
 nevow.flat.flatstan.RendererSerializer            nevow.inevow.IRenderer
 nevow.flat.flatstan.DirectiveSerializer           nevow.stan.directive
 nevow.flat.flatstan.SlotSerializer                nevow.stan.slot
+nevow.flat.flatstan.CachedSerializer              nevow.stan.cached 
 nevow.flat.flatstan.ContextSerializer             nevow.context.WovenContext
 nevow.flat.flatstan.DeferredSerializer            twisted.internet.defer.Deferred
 nevow.flat.flatstan.DeferredSerializer            twisted.internet.defer.DeferredList
Index: Nevow/nevow/flat/flatstan.py
===================================================================
--- Nevow/nevow/flat/flatstan.py	(revision 1192)
+++ Nevow/nevow/flat/flatstan.py	(working copy)
@@ -9,10 +9,14 @@
 from nevow import util
 from nevow.stan import Proto, Tag, xml, directive, Unset, invisible
 from nevow.inevow import IRenderer, IRendererFactory, IGettable, IData
-from nevow.flat import precompile, serialize
+from nevow.flat import precompile, serialize, iterflatten
 from nevow.accessors import convertToData
 from nevow.context import WovenContext
 
+from time import time as now
+from cStringIO import StringIO
+from twisted.internet import defer
+
 allowSingleton = ('img', 'br', 'hr', 'base', 'meta', 'link', 'param', 'area',
                   'input', 'col', 'basefont', 'isindex', 'frame')
 
@@ -226,6 +230,45 @@
         return serialize(original.default, context)
     return serialize(data, context)
 
+_CACHE = {}
+def CachedSerializer(original, context):
+    cached = _CACHE.get(original.name, None)
+    _now = now()
+    life = _now-original.lifetime
+    if cached and (cached[0] > life or not original.lifetime):
+        yield cached[1]
+        return
+    io = StringIO()
+    for child in iterflatten(original.children, context, io.write,
+                             lambda item: True):
+        if isinstance(child, tuple):
+            childDeferred, childReturner = child
+ 
+            d = defer.Deferred() ## A new deferred for the outer loop, whose result
+            ## we don't care about, because we don't want the outer loop to write
+            ## anything when this deferred fires -- only when the entire for loop
+            ## has completed and we have all the "children" flattened
+ 
+            def innerDeferredResultAvailable(result):
+                childReturner(result) ## Cause the inner iterflatten to continue
+                d.callback('') ## Cause the outer iterflatten to continue
+                return ''
+ 
+            childDeferred.addCallback(innerDeferredResultAvailable)
+ 
+            ## Make the outer loop wait on our new deferred.
+            ## We call the new deferred back with ''
+            ## Which will cause the outer loop to write '' to the request,
+            ## which doesn't matter. It will then call our "returner",
+            ## which is just the noop lambda below, because we don't care
+            ## about the return result of the new deferred, which is just
+            ## ''
+ 
+            yield d, lambda result: ''    
+    result = io.getvalue()
+    _CACHE[original.name] = (_now, result)
+    yield result
+
 def ContextSerializer(original, context):
     originalContext = original.clone(deep=False)
     originalContext.precompile = context and context.precompile or False
Index: Nevow/nevow/stan.py
===================================================================
--- Nevow/nevow/stan.py	(revision 1192)
+++ Nevow/nevow/stan.py	(working copy)
@@ -119,8 +119,33 @@
         """
         raise NotImplementedError, "Stan slot instances are not iterable."
 
+class cached(object):
+    """Marker for cached content
+    """
+    __slots__ = ['name', 'children', 'lifetime']
 
+    def __init__(self, name, lifetime=0):
+        self.name = name
+        self.children = []
+        self.lifetime = lifetime
 
+    def __repr__(self):
+        return "cached('%s','%s')" % self.name, self.lifetime
+
+    def __getitem__(self, children):
+        """cached content is what is being cached
+        """
+        if not isinstance(children, (list, tuple)):
+            children = [children]
+        self.children.extend(children)
+        return self
+
+    def __iter__(self):
+        """Prevent an infinite loop if someone tries to do
+            for x in cached('foo'):
+        """
+        raise NotImplementedError, "Stan slot instances are not iterable."
+
 class Tag(object):
     """Tag instances represent XML tags with a tag name, attributes,
     and children. Tag instances can be constructed using the Prototype
Index: Nevow/nevow/rend.py
===================================================================
--- Nevow/nevow/rend.py	(revision 1192)
+++ Nevow/nevow/rend.py	(working copy)
@@ -30,6 +30,7 @@
 from nevow import flat
 from nevow.util import log
 from nevow import util
+from nevow import url
 
 import formless
 from formless import iformless
@@ -374,6 +375,7 @@
             self.children = {}
         self.children[name] = child
     
+_CACHE = {}
 
 class Page(Fragment, ConfigurableFactory, ChildLookupMixin):
     """A page is the main Nevow resource and renders a document loaded
@@ -384,12 +386,48 @@
 
     buffered = False
 
+    cacheTimeout = None # 0 means cache forever, >0 sets the seconds of caching
+    __lastCacheRendering = 0 # this should not be touched by the parent class
+
     beforeRender = None
     afterRender = None
     addSlash = None
 
     flattenFactory = flat.flattenFactory
 
+    def hasCache(self, ctx):
+        if self.cacheTimeout is None:
+            return None
+
+        _now = now() # run gettimeofday only once
+        timeout = _now > self.__lastCacheRendering + self.cacheTimeout and \
+                  self.cacheTimeout > 0
+        c = self.lookupCache(ctx)
+        if timeout or c is None:
+            self.__lastCacheRendering = _now # stop other renders
+            from twisted.internet.defer import Deferred
+            d = Deferred()
+            self.storeCache(ctx, d)
+            # force only this rendering, others will wait the deferred
+            c = None
+        return c
+    def chainDeferredCache(self, ctx, d):
+        if self.cacheTimeout is None:
+            return d
+
+        from twisted.internet.defer import Deferred
+        c = self.lookupCache(ctx)
+        if isinstance(c, Deferred):
+            # we're the thread that went ahead to refresh the cache
+            d.chainDeferred(c)
+        return d
+    def cacheIDX(self, ctx):
+        return str(url.URL.fromContext(ctx))
+    def storeCache(self, ctx, c):
+        _CACHE[self.cacheIDX(ctx)] = c
+    def lookupCache(self, ctx):
+        return _CACHE.get(self.cacheIDX(ctx))
+
     def renderHTTP(self, ctx):
         ## XXX request is really ctx now, change the name here
         request = inevow.IRequest(ctx)
@@ -411,11 +449,18 @@
             if self.afterRender is not None:
                 self.afterRender(ctx)
 
-        if self.buffered:
+        c = self.hasCache(ctx)
+        if c:
+            finishRequest()
+            return c
+
+        if self.buffered or self.cacheTimeout is not None:
             io = StringIO()
             writer = io.write
             def finisher(result):
-                request.write(io.getvalue())
+                c = io.getvalue()
+                self.storeCache(ctx, c)
+                request.write(c)
                 finishRequest()
                 return result
         else:
@@ -427,7 +472,7 @@
         doc = self.docFactory.load()
         ctx =  WovenContext(ctx, tags.invisible[doc])
 
-        return self.flattenFactory(doc, ctx, writer, finisher)
+        return self.chainDeferredCache(ctx, self.flattenFactory(doc, ctx, writer, finisher))
 
     def rememberStuff(self, ctx):
         Fragment.rememberStuff(self, ctx)
Index: Nevow/nevow/vhost.py
===================================================================
--- Nevow/nevow/vhost.py	(revision 1192)
+++ Nevow/nevow/vhost.py	(working copy)
@@ -19,7 +19,7 @@
 """
 
     def getStyleSheet(self):
-        return self.stylesheet
+        return VirtualHostList.stylesheet
  
     def data_hostlist(self, context, data):
         return self.nvh.hosts.keys()
Index: Nevow/nevow/static.py
===================================================================
--- Nevow/nevow/static.py	(revision 1192)
+++ Nevow/nevow/static.py	(working copy)
@@ -271,7 +271,7 @@
             return self.redirect(request)
 
         #for content-length
-        fsize = size = self.getFileSize()
+        fsize = csize = self.getFileSize()
 
         request.setHeader('accept-ranges','bytes')
 
@@ -302,20 +302,20 @@
                        "Syntactically invalid http range header!"
                 start, end = string.split(bytesrange[1],'-')
                 if start:
-                    f.seek(int(start))
+                    start = int(start)
+                    f.seek(start)
                 if end:
-                    end = int(end)
-                    size = end
+                    csize = int(end) - start + 1
                 else:
-                    end = size
+                    csize -= start
                 request.setResponseCode(http.PARTIAL_CONTENT)
                 request.setHeader('content-range',"bytes %s-%s/%s " % (
-                    str(start), str(end), str(size)))
+                    str(start), str(end), str(fsize)))
                 #content-length should be the actual size of the stuff we're
                 #sending, not the full size of the on-server entity.
                 fsize = end - int(start)
 
-            request.setHeader('content-length', str(fsize))
+            request.setHeader('content-length', str(csize))
         except:
             traceback.print_exc(file=log.logfile)
 
@@ -323,7 +323,7 @@
             return ''
 
         # return data
-        FileTransfer(f, size, request)
+        FileTransfer(f, csize, request)
         # and make sure the connection doesn't get closed
         return request.deferred
 
@@ -354,17 +354,16 @@
         self.file = file
         self.size = size
         self.request = request
-        self.written = self.file.tell()
         request.registerProducer(self, 0)
 
     def resumeProducing(self):
         if not self.request:
             return
-        data = self.file.read(min(abstract.FileDescriptor.bufferSize, self.size - self.written))
+        data = self.file.read(min(abstract.FileDescriptor.bufferSize, self.size))
         if data:
-            self.written += len(data)
             self.request.write(data)
-        if self.file.tell() == self.size:
+            self.size -= len(data)
+        if self.size <= 0:
             self.request.unregisterProducer()
             self.request.finish()
             self.request = None



More information about the Twisted-web mailing list