[Twisted-web] Re: Serving files from many web-servers thru one central web-server

Wed Sep 29 00:40:47 MDT 2004

Thomas Weholt <thomas.weholt at gmail.com> writes:

> Oh, darn!! Forgot all about this when I started my project. I've read
> thru most of the docs I've  found so far, but I'm still somewhat
> clueless. Can anybody provide a simple example of how to do this,
> preferrably without using the examples in
> http://twistedmatrix.com/documents/current/howto/using-twistedweb#auto19.
> I'm hooking a xmlrpc-handler and a UDP listener into it as well and
> find the good ol'
(...)

Well, I experimented tonight and here's a quick example of some
situations.  In re-reading your original post, I realized it wasn't
clear if your "reads data from several subnet servers" comment
referred to wanting to access web resources on the internal servers,
or just some other service you needed data from.  If the latter, then
you can use your own PB session that can be richer in interface than a
web resource if you wanted, so I gave a simple example of that too.

This example just runs both the simulated main server and subset
server objects within the same process over a loopback connection, but
should work identically over any other link.

There is a main and child resource on the server side, and the same on
the internal/subnet side, tied into a URL on the server side by using
the ResourcePublisher/ResourceSubscription.  An additional server side
resource turns into its own pure PB call to a remote server object.
Once running you can access the following URLs:

http://localhost:8000                    ExternalRoot
http://localhost:8000/child              ExternalChild
http://localhost:8000/data               InternalData.remote_getData()
http://localhost:8000/internal           InternalRoot
http://localhost:8000/internal/child     InternalChild

In the InternalData case, I'm making a new PB session for each
rendering request.  Presumably you'd want to structure things to
maintain a persistent connection only reconnecting when necessary
(which, BTW, is basically what ResourceSubscription does).

It all seems to work as expected... it's simplistic but hopefully
it'll point you in the right direction.  Shouldn't be any problem to
tie in additional protocols (such as XMLRPC/UDP) into either the main
server or subnet server processes.

-- David

          - - - - - - - - - - - - - - - - - - - - - - - - -

import sys

from twisted.python import log
from twisted.internet import reactor
from twisted.spread import pb
from twisted.web import server, resource, distrib

#
# An internal PB server object on the internal subnet server, with simple
# direct access (no authentication) via a root object.

class InternalData(pb.Root):

    def remote_getData(self):
        # This could be its own deferrable operation
        return 'Internal data'

#
# Resources on the internal subnet servers
#

class InternalChild(resource.Resource):
    """A child resource rendered on the internal server"""

    def render(self, request):
        return '<html><body>Internal Child</body></html>'

class InternalRoot(resource.Resource):
    """The root of the tree rendered on the internal server"""

    def getChild(self, path, request):
        # Support direct rendering (no trailing "/" on request)
        if path == '':
            return self
        else:
            return resource.Resource.getChild(self, path, request)

    def render(self, request):
        return '<html><body>Internal Root</body></html>'

#
# Resources on the primary web server
#

class MainData(resource.Resource):
    """A child resource that renders a data call to the internal server"""

    def __init__(self, host, port):
        resource.Resource.__init__(self)
        self.host = host
        self.port = port

    def render(self, request):
        """Make a request to the remote root object, and use that result
        as the result of our rendering"""

        def failure(value):
            request.write('<html><body>'
                          'Unable to access data:<br>%s'
                          '</body></html>' % value)

        def success(value):
            request.write('<html><body>%s</body></html>' % value)

        # Right now we make a new connection to the internal host for
        # each rendering request (pretty darn inefficient!)
        factory = pb.PBClientFactory()
        reactor.connectTCP(self.host, self.port, factory)

        # Obtain a reference to the remote object, call the getData method
        # and then disconnect.
        root = factory.getRootObject()
        root.addCallback(lambda root:
                         root.callRemote('getData').addCallback(success))
        root.addErrback(failure)
        root.addCallback(lambda _: request.finish())
        root.addCallback(lambda _: factory.disconnect())
        return server.NOT_DONE_YET

class MainChild(resource.Resource):
    """A child resource rendered directly on the main server"""

    def render(self, request):
        return '<html><body>Main Server Child</body></html>'

class MainRoot(resource.Resource):
    """The primary root resource on the main server"""

    def getChild(self, path, request):
        # Support direct rendering (no trailing "/" on request)
        if path == '':
            return self
        else:
            return resource.Resource.getChild(self, path, request)

    def render(self, request):
        return '<html><body>Main Server Root</body></html>'

#
# Simulate main and subnet servers.  The main server will listen on port
# 8000 and the subnet server will listen (for PB connections) on port 8001.
# Additionally the subnet server will provide the InternalData object
# on port 8002.
#

if __name__ == "__main__":

    #
    # Build up a subnet server "site":
    #     /         InternalRoot
    #     /child    InternalChild
    #
    iroot = InternalRoot()
    iroot.putChild('child', InternalChild())
    isite = server.Site(iroot)

    #
    # Build up the main server "site":
    #     /          MainRoot
    #     /child     MainChild
    #     /data      Render result of call to InternalData's retrieveData
    #     /internal  Request to / on subnet server
    #
    root = MainRoot()
    root.putChild('child', MainChild())
    root.putChild('data', MainData('localhost', 8002))
    root.putChild('internal', distrib.ResourceSubscription('localhost',8001))
    site = server.Site(root)

    #
    # Now start both servers listening.  Note that if these were really
    # running on separate machines, the internal server could do a listenTCP
    # for isite on 8000 to support normal web lookups, while also supporting
    # port 8001 for the PB proxied lookups.
    #
    reactor.listenTCP(8000, site)
    reactor.listenTCP(8001,
                      pb.PBServerFactory(distrib.ResourcePublisher(isite)))
    reactor.listenTCP(8002,pb.PBServerFactory(InternalData()))

    log.startLogging(sys.stdout)
    reactor.run()