[Twisted-web] Re: Serving files from many web-servers thru one central web-server

[David Bolen]

Thomas Weholt <thomas.weholt at gmail.com> writes:

> A few users connect to the main server, requesting files on the
> subnet web-servers. The main server reads data from several subnet
> servers and writes the data back to the requesting clients. How can I
> do this in twisted, without blocking, and handle several clients?
> We're not talking hardcore P2P here with thousands of clients, most
> likely 2-5 concurrent users, 10 at the most.

I haven't had an opportunity to use it myself yet, but there is a
twisted.web.spread module that may work since all of your servers are
using Twisted.

On the subnet servers, wrap your site object in the ResourcePublisher
object (a pb.Root subclass) and set that up to listen on an
appropriate port.

On the main server, for each resource root that you want proxied out
to a remote server, insert an appropriate child resource using
ResourceSubscription.

A PB link used to transmit the requests between the two machines,
which means that all of the server render() calls to be proxied are
handled in a deferred fashion.

I expect there may be a way to interconnect your server into a client
HTTP class to proxy to the subset servers using a more traditional web
request, but given that you are Twisted throughout, probably no reason
not to go ahead and use the PB approach.

It does look like these classes still use the older pb.getObjectAt
approach for the connection rather than the newer getRootObject, but
it should still work.  If it doesn't directly suit what you want, it
it may at least give you an idea for your own approach (since neither
of those classes is overly complex).

-- David