[Twisted-Python] Returning a deferred from buildProtocol t.i.p.Factory

[Tom van Neerijnen]

Thanks for both those suggestions.
I'll be taking a closer look at txLoadbabancer when I get time as it looks
like it'll take care of a lot of my desired functionality out the box.
To get started tho I'll move my async routing decision call into the
protocol as suggested.

Is there any reason why the internal calls to buildProtocol shouldn't be
wrapped in a maybeDeferred?


On Sat, Nov 16, 2013 at 7:05 PM, Lucas Taylor <ltaylor.volks at gmail.com>wrote:

>
> On Nov 16, 2013, at 7:09 AM, Tom van Neerijnen wrote:
>
> Hi all
>
> I'm building a simple TCP load balancer based on a code snippet from Glyph
> on SO:
> http://stackoverflow.com/questions/4096061/general-question-regarding-wether-or-not-use-twisted-in-tcp-proxy-project
>
> It's served me well but I can't work out how to convert Glyphs round robin
> retrieval of the server endpoint into an async balancing decision in the
> buildProtocol method of the Factory. If I return a deferred here it fails
> with an AttributeError: Deferred instance has no attribute 'makeConnection'.
>
> Currently I'm working around this by running a separate management loop
> that periodically updates a dictionary with all the data necessary to make
> my routing decision so that I can do it without a deferred. This worries me
> because I may be making my decision on slightly stale data and I'd really
> like this to be a real time decision as the connection comes in. Does
> anyone have a clever way of doing this?
>
>
>
> Hi Tom,
>
> One possibly unexpected aspect of using @inlineCallbacks is that the
> decorated function itself returns a Deferred. This is why you see the
> AttributeError...the machinery calling buildProtocol expects an IProtocol
> instance (or None), and the function is returning a Deferred.
> `defer.returnValue()` is provided to the callback on that Deferred, not as
> a direct return value from the decorated function.
>
> If you want to make the routing decision when the client connects, then
> you could push the decision-making process down into the Protocol itself.
>
> Here's a quick mockup overriding connectionMade in a ProxyServer protocol
> subclass. It calls the factory routing function (which may or may not
> return a deferred), and connects the proxy once the decision has been made.
>
>
> from twisted.internet.protocol import Factory
> from twisted.protocols.portforward import ProxyServer
>
>
> class Balancer(Factory):
>     protocol = RoutingProxyServer
>     routing_func = port_routing_decision_async
>
>
> class RoutingProxyServer(ProxyServer):
>
>     def connectionMade(self):
>         # Don't read anything from the connecting client until we have
>         # somewhere to send it to.
>         self.transport.pauseProducing()
>
>         client = self.clientProtocolFactory()
>         client.setServer(self)
>
>         if self.reactor is None:
>             from twisted.internet import reactor
>             self.reactor = reactor
>
>         def connectProxy(host, port):
>             self.reactor.connectTCP(host, port, client)
>
>         d = maybeDeferred(self.factory.routing_func)
>         d.addCallback(connectProxy)
>         d.addErrback(log.err)
>
>
> Lucas
>
>
>
>
>
> An example is below. The hashed out buildProtocol is a synchronous
> decision which works. Thanks in advance!
>
> from twisted.internet.protocol import Factory
> from twisted.protocols.portforward import ProxyFactory
> from twisted.internet import reactor, defer
> import random
>
> from twisted.python import log
> import sys
> log.startLogging(sys.stderr)
>
> local_ports = set([1024, 1025])
>
> def port_routing_decision_sync():
>     return random.choice(list(local_ports))
>
> def port_routing_decision_async():
>     d = defer.Deferred()
>     reactor.callLater(1, d.callback, port_routing_decision_sync())
>     return d
>
> class Balancer(Factory):
>     # def buildProtocol(self, addr):
>     #     port = port_routing_decision_sync()
>     #     print "connecting to local port {}".format(port)
>     #     return ProxyFactory("127.0.0.1", port).buildProtocol(addr)
>
>     @defer.inlineCallbacks
>     def buildProtocol(self, addr):
>         port = yield port_routing_decision_async()
>         print "connecting to local port {}".format(port)
>         defer.returnValue(ProxyFactory("127.0.0.1",
> port).buildProtocol(addr))
>
> def main():
>     factory = Balancer()
>     reactor.listenTCP(5678, factory)
>     reactor.run()
>
> if __name__ == "__main__":
>     main()
>
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20131117/96725b12/attachment.html>