[Twisted-Python] Deferred Network requests

[Phil Mayers]

Simon Pickles wrote:
> Hi,
> 
> I've been using twisted in a low-level way for some time and am now 
> interested in the higher level aspects.
> 
> Particularly I hope to be able to make deferred requests across a 
> network. I'd like to have a hub server, with many small apps (modules) 
> which are all clients of the hub server.
> 
> I'd like the modules to be able to request information from each other, 
> via the hub, in a deferred way, so the callback will be triggered when 
> the requested information arrives.
> 
> So I need a chain like this:
> 
> ModuleA.RequestName(id).addCallback(modACallback) -> Hub
> Hub.RequestName(id).addCallback(hubCallback)-> ModuleB
> ModuleB.SendNameToHub() -> Hub
> Hub.hubCallback triggered: SendNameToModA() -> ModuleA
> ModuleA.modACallback triggered - request is complete.
> 
> Perhaps one issue is that I have a client->server->client sequence.
> 
> Although it looks slow, I am designing my system to be as concurrent as 
> possible. Also, the hub can make decisions about sharing requests with 
> several modules on a round-robin, or sending requests (and events, 
> particularly) to more than one module.
> 
> I know I can do this through TCP or similar, but hoped someone might 
> suggest which elements of twisted (a very large framework) I should look 
> at more closely.

Perspective broker (pb) would work, provided the values you are passing 
round are plain python types e.g. int, str, list, dict, tuple.

You'd have something like:

from twisted.spread import pb
from twisted.internet import reactor

class Module(pb.Referenceable):
   def __init__(self, name, server, port=1234):
     self.f = pb.PBClientFactory()
     reactor.connectTCP(server, port, f)
     self.f.getRootObject().addCallback(self.signon, name)

   def signon(self, root, name):
     self.root = root
     self.root.callRemote('signon', name, self)

   # methods for use by remote
   def remote_doThing(self):
     return aDeferredThing()

   # methods for use by local process
   def requestName(self, id):
     return self.root.addCallback(requestName2, id)
   def requestName2(self, root, id):
     return root.callRemote('requestName', id)

ModuleA = Module('ModuleA', server)
reactor.run()

...and on the server site:

from twisted.spread import pb

class Server(pb.Root):
   modules = {}

   # function a module registers with
   def remote_signon(self, name, module):
     if name in self.modules:
         self.modules[name].append(module)
     else:
         self.modules[name] = [module]

   # proxy to a copy of ModuleA, with load-balancing
   def remote_modulea_doThing(self, arg):
     modulea_list = self.modules['ModuleA']
     mod = random.choice(modulea_list)
     return mod.callRemote('doThing', arg)

   # proxy wherever...
   def remote_requestName(self, id):
     mod = ...
     return mod.requestName(id)

The above is very hacky - the class-global dict() is wrong, and there is 
no handling of PB disconnection errors and so forth, but the basic idea 
is sound.

> 
> Many thanks!
> 
> Simon
>