[Twisted-Python] Guidance on Proxy-type Application

[Aaron Bush]

glyph,

Sorry for the delayed response.  I wanted to thank you for providing such a
complete example.  This is excellent.  Now I just need to wrap by head
around how it all goes together and works.

The logic of the queue append and pop of a deferred in ProxyClient
forwardLine and lineRecieve are throwing me for a loop.  I'll keep tracing
it and get it down.

Thanks Again!
-ab

On Thu, May 28, 2009 at 6:58 PM, <glyph at divmod.com> wrote:

>
> On 01:23 am, asb.bush at gmail.com wrote:
>
>> I have just started to look at the Twisted framework and would like to put
>> it
>> to use for a new project I am working on.  Not being very familiar with
>> the
>> framework and fairly new to Python in general I would like to ask a
>> design/architecture question.  (I have written similar applications in C
>> but
>> would prefer to start this in the right direction and not write Python
>> like
>> C.)
>>
>
> Thanks for asking!
>
> I apologize for the delay in my answer.  I started writing up a simple
> example (attached) but was discouraged to find that it was 100 lines long
> and required too much explaining.
>
> Then I started documenting it and explaining every line but that was a very
> long, tedious message.  So, it doesn't have much in the way of explanation;
> I hope you will find it useful regardless.
>
>> The application has the following model:
>>
>
>  Many clients connect to the Application and prefer to leave the connection
>> open.  They will send messages across this connection.  They will expect
>> to
>> get a message back at some point later, they do not wait for a response
>> (async).  The clients are already coded (legacy) and just need to send
>> their
>> proprietary protocol to the new Application (written using Twisted).
>>
>
> This is *almost* a FAQ.  At least, you may find this to be a useful answer:
>
> <
> http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#HowdoImakeinputononeconnectionresultinoutputonanother
> >
>
>> The Twisted application will take the data from the clients and do some
>> transformation on it then send the message on to another server (3rd
>> party).
>> This connection to "another" server must be a single connection, not one
>> connection per client.  This connection should also be persistent and not
>> opened/closed for each client message sent.  Ideally if the 3rd party
>> server
>> is down then I would also not accept client connections as the messages
>> are
>> time sensitive and should not be stored and forwarded.  At some point the
>> 3rd
>> part will send a message back and the Application will route it back to
>> the
>> original source.  Basically request/reply pattern.
>>
>
> The example that I've attached does basically this.  Run it and then run
> 'telnet localhost 4322', and type some lines; you will see that they are
> transformed and echoed back to you, both by the proxy and by the protocol
> being proxied.
>
> At a high level, the answer to your question is so simple that it's hard to
> express.  Basically, you just need to have all the relevant objects having
> references to each other, and calling methods to achieve the desired effect.
>  The less magic, the better.
>
> More precisely, you need an object responsible for managing your outgoing
> connections to your legacy server, so that it can handle disconnection and
> reconnection, queueing messages and so on.  Then you need your proxy server
> factory to hold a reference to that object, so that it can create references
> from each proxy server protocol connection object to the connection manager.
>
> This is related to another recent thread - you can see my message in that
> thread here:
>
>   http://thread.gmane.org/gmane.comp.python.twisted/18377/focus=18385
>
>> I have been reading through the archives and the twisted docs and have
>> also
>> looked over the Hex-dump port-forwarding recipe but not found anything
>> that
>> explains how to use twisted for this model.  Hex-dump is close but
>> opens/closes the connection to the server on each client connection.
>>
>
> I'm not sure why hex-dump port-forwarding is particularly relevant to this
> example.  Is it just because this is an application that connects from one
> host to another?
>
>> I am thinking that there will be two Factories [and two protocols: 1) for
>> clients and 2) for 3rd party].  I am not sure how to best establish both
>> the
>> listening factory and the client to 3rd party factory.  Once they are
>> established what is the preferred way in Twisted to pass a message from
>> one
>> protocol to another?
>>
>
> This part of your question is almost exactly the FAQ I mentioned above :).
>  To reiterate that answer, you just need to have references between objects,
> and call methods on the objects you want to do stuff.
>
> If you have a client connection object, just get a reference to that from
> the relevant server connection object and call methods on the client object
> to emit messages on the client protocol, handling any responses
> appropriately.  Deferreds can help with that latter part.
>
> It is always better if you can establish that reference as simply as
> possible; for example, by passing parameters to the __init__ of various
> classes.  Again, for reasons that have nothing to do with Twisted
> specifically, it's a bad idea to try to establish these references by having
> global variables floating around.
>
> Here's a very very simple example of the "good way" to propagate some data
> to protocol instances that need it:
>
>   class MyProtocol(Protocol):
>       def __init__(self, data):
>           self.data = data
>
>   class MyFactory(Factory):
>       def __init__(self, data):
>           self.data = data
>
>       def buildProtocol(self, addr):
>           return MyProtocol(self.data)
>
>   reactor.listenTCP(8765, MyFactory("some data"))
>
> and here's a simple example of a really bad way (don't do this!):
>
>   class MyProtocol(Protocol):
>       def connectionMade(self):
>           self.bleh = bleh
>
>   bleh = "some data"
>   f = Factory()
>   f.protocol = MyProtocol
>   reactor.listenTCP(9876, f)
>
> Even in C, I'm pretty sure it's better style to pass structures to
> functions than to abuse piles of local variables :).  I only illustrate this
> bad style here because it seems to be a common antipattern.  The Protocol
> class itself doesn't take any parameters to __init__, and Twisted's users
> don't always realize that protocols and factories and so on are just regular
> objects, with no special rules; they just get methods called on them by the
> reactor.
>
>> Any pointers or sample code that you can offer is greatly appreciated. I
>> would really like to cook this in Twisted and not go back to the C way.
>>
>
> Based on what you've said so far, I think you're basically on the right
> track.  Good luck!
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20090602/ef3801aa/attachment.html>