[Twisted-Python] Can transport.write() to hostname instead of IP address?

[Sean DiZazzo]

>
> To summarize, you are writing a client application using Twisted which
> needs to talk with a service behind an Nginx reverse proxy. The reverse
> proxy requires use of TLS (a.k.a. SSL) and SNI to identify the appropriate
> backend service.


Why was that so difficult for me?  lol.

This is all great, Tom.  Thank you.  I guess I didn't find this because up
until now I have only played around with endpoints.  The twisted servers I
have running are all using the older interfaces for setting up and handling
connections.  I balked at learning endpoints after struggling to figure out
Twisted the first time.  I guess I'm a holdout.  I had barely even looked
at python 3 until the last few months.  Now I love it.  ;P  Its time for
endpoints to get their turn in the sun.

It will take some rethinking and rewriting, but at least now I have a way
forward.  Thank you!

On Thu, Aug 16, 2018 at 8:18 PM, Tom Most <twm at freecog.net> wrote:

> Hi Sean,
>
> To summarize, you are writing a client application using Twisted which
> needs to talk with a service behind an Nginx reverse proxy. The reverse
> proxy requires use of TLS (a.k.a. SSL) and SNI to identify the appropriate
> backend service.
>
> If you were using HTTP, Twisted's twisted.web.client.Agent
> <https://twistedmatrix.com/documents/current/api/twisted.web.client.Agent.html>
> API already does the right thing here -- it takes the hostname from the URL
> it is given and populates SNI.
>
> For your custom protocol on top of TLS, you should use a TLS endpoint to
> connect to the server from your client. This is a little difficult to
> discover because there isn't a TLS endpoint per se -- instead, there is a
> function which wraps another endpoint. See the TLS section in the
> endpoint documentation
> <https://twistedmatrix.com/documents/current/core/howto/endpoints.html#endpoint-types-included-with-twisted>,
> which includes this example:
>
> wrapped = HostnameEndpoint('example.com', 443)contextFactory = optionsForClientTLS(hostname=u'example.com')endpoint = wrapClientTLS(contextFactory, wrapped)conn = endpoint.connect(Factory.forProtocol(Protocol))
>
>
> I'll break this down:
>
> 1. HostnameEndpoint will resolve the hostname to an IP address and creates
> a TCP connection to port 443.
> 2. The optionsForClientTLS
> <https://twistedmatrix.com/documents/18.7.0/api/twisted.internet.ssl.optionsForClientTLS.html>
> function generates an object which represents the TLS connection options.
> Importantly, it enables SNI based on the hostname passed to it.
> 3. wrapClientTLS
> <https://twistedmatrix.com/documents/current/api/twisted.internet.endpoints.html#wrapClientTLS>
> returns an endpoint which layers TLS on top of the plain TCP connection
> generated by HostnameEndpoint. It also takes the TLS options as an argument.
> 4. conn is a Deferred which will fire with a protocol instance generated
> by the factory passed to connect().
>
> This is basically what Agent does internally, as I understand it.
>
> Hope this helps,
> Tom
>
> On Thu, Aug 16, 2018, at 6:44 PM, Sean DiZazzo wrote:
>
> I guess thats still kind of confusing without making something more
> clear...
>
> In my example, both myprotocol.example.com and test.example.com DNS
> records would point to the same IP address.  One nginx instance then
> listens on that IP and serves up several ssl apps. They go through a
> "mapper" that uses the SNI and the ssl_preread directive to read the
> destination hostname of the packet to determine which app to route the
> traffic to.
>
> I just want transport.write() to not resolve the ip address of the host I
> pass in.  Everything will work if it connects and sends packets to
> myprotocol.example.com:443 instead of 23.23.23.23:443.
>
> Nginx reference:
> http://nginx.org/en/docs/stream/ngx_stream_ssl_preread_module.html
>
> On Thu, Aug 16, 2018 at 6:14 PM, Sean DiZazzo <sean.dizazzo at gmail.com>
> wrote:
>
> Thanks for responding, Adi!
>
> I don't want each packet to go it's own way from Twisted.  They all go to
> the same place from each instance of the server/protocol.  They go to my
> custom protocol listening on another local port.
>
> It's just that I'm serving up several different ssl apps on the same nginx
> server, and nginx uses the hostname to route the packets.  So in this case,
> traffic coming in on http.example.com:443 might be routed to an https app
> listening on a socket, and traffic coming in to myprotocol.example.com:443
> should be routed to my own protocol listening on port 9999.  So if nginx
> doesn't get the hostname, it doesn't know to route the packet to my custom
> protocol instead of the web server.  Does that make sense?
>
> It seems that the transport is resolving the hostname to an ip address and
> then sending the traffic to the generic ip which is not enough info for
> nginx to route the packet correctly.
>
> On Thu, Aug 16, 2018 at 5:49 PM, Adi Roiban <adi at roiban.ro> wrote:
>
> On Fri, 17 Aug 2018 at 01:25, Sean DiZazzo <sean.dizazzo at gmail.com> wrote:
> >
> > Hi all!
> >
> > After I start a reactor connecting to a specific hostname and port, I do
> my thing and then call transport.write() to send the data to the peer.
> >
> > From what I can tell, though, the hostname is resolved, and the data is
> written back to the ip address itself, instead of the hostname I started
> the reactor with.
> >
> > This is a problem in my case because we are using nginx's ssl_preread
> server_name directive to route several different streams all coming in on
> the same ip address.
> >
> > So the write() method needs to explicitly use the hostname to route the
> packet properly.
> >
> > So... Is there any way to have transport.write() use the hostname given
> instead of it's resolved IP address? Or am I missing something?
> >
>
> I assume you are using TCP here.
>
> I guess that you are missing something.
>
> If you want each write to go over its own way / route and have the
> hostname re-resolved you should open + write + close a connection for
> each write.
>
> But I think that there is something else there and this is now what you
> want :)
> Do you use HTTP or have a custom protocol?
>
> Cheers,
>
> Adi Roiban
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
> *_______________________________________________*
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20180816/9bf5c311/attachment-0002.html>