[Twisted-web] internal vs. external hostname in Request.getHost9)

[Ilya Skriblovsky]

Tickets you have mentioned and forwarded-for-5807 branch are mostly about
parsing X-Forwarded-For in order to obtain correct client IP. While it is
valuable task, it is not what strikes me right now.

I'm now more concerned with an absence of API for getting user-visible
server's name, not client's ip.

Look, I'm currently porting my app from Django to Klein and noticed strange
behavior of Klein. For example:
@app.route('/alias', alias=True)
@app.route('/path')
def path(request): return b'42'

When /alias is requested werkzeug generates a redirect to /path. But Klein
is passing Request.getHost() to Werkzeug, so redirect gets internal
hostname and exposes backend's internal hostname and port to the user.
Seems like Klein is passing incorrect hostname to Werkzeug. But how can we
fix that?

There are two methods in Request:
• Request.getHost() — "Get my originally requesting transport's host" as
doc says. Ok, seems like this method intentionally returns server's
internal address.
• Request.getRequestHostname() —doc says:
>> "Get the hostname that the user passed in to the request. This will
either use the Host: header (if it is available) or the host we are
listening on if the header is unavailable."
Cool, but why does this method only returns a hostname without a port? It
intentionally strips out the port number from Host header. What is the
point of such implementation? This method is used only a couple of times
inside Twisted itself, and in both places Twisted gets what
getRequestHostname() returned and mixes it with request.getHost().port
which is *definitely* incorrect, because the former is user-visible while
latter is internal. So if my backend server is using different port than a
fronend, it is impossible to use getRequestHostname() to build user-visible
URL. I think current getRequestHostname() implementation is broken.

So I have two proposals:

Proposal #1 (fixing current behavior):
• Variant #1: Change Request.getRequestHostname() to return
b"hostname:port". I think this is the correct thing to do, but this is a
backward-incompatible change.
- or -
• Variant #2: Change Klein to use Request.getHeader(b'Host') with fallback
to Request.getHost()

Proposal #2 (adding new feature if Variant #1 is choosed):
• Add useXForwardedHost=False argument to Request.getRequestHostname() and
useXForwardedProto=False to Request.isSecure(). If True is passed, these
methods will obey corresponding request headers that are de-facto standard
for reverse proxies. Also add corresponding options to Klein app. This can
simplify reverse proxy configuration a bit.

-- ilya

вт, 14 мар. 2017 г. в 10:33, Ilya Skriblovsky <ilyaskriblovsky at gmail.com>:

> Thanks, I will study tickets you mentioned and hopefully fix it.
> Quick-n-dirty fix gave me only two failed tests and in both cases it seems
> to be a wrong assumption in tests. So I hope this change won't break the
> world.
>
> -- ilya
>
> вт, 14 мар. 2017 г. в 10:12, Glyph Lefkowitz <glyph at twistedmatrix.com>:
>
>
> On Mar 13, 2017, at 11:01 PM, Ilya Skriblovsky <ilyaskriblovsky at gmail.com>
> wrote:
>
> Hi,
>
> I'm using Twisted Web server behind Nginx reverse-proxy and I'm getting
> backend's internal host:port from Request.getHost().
>
> Seems like Request.host is explicitly set to socket's address (i.e.
> internal address) here: ​
> https://github.com/twisted/twisted/blob/trunk/src/twisted/web/http.py#L838
> But comment at ​
> https://github.com/twisted/twisted/blob/trunk/src/twisted/web/http.py#L1297
> and what this method does points that Request.host meant to reflect Host
> header of the request, i.e. user-visible hostname and port.
>
> This creates problems for me when using Klein because it correctly uses
> Request.getHost() to create host part of URLs for redirects.
>
> It seems like inconsistency in Twisted code. I'd expect Request.host
> should be only set from the Host request header to reflect user-visible
> hostname, not the internal backend server's address. Or may be I'm missing
> something?
>
>
> You're absolutely correct!  I even filed a ticket for this functionality,
> 5 years ago: https://twistedmatrix.com/trac/ticket/5807  There's even a
> branch for it.  Oddly enough we *do* have a *private*
> _XForwardedForRequest, but... it's only used for logging, for some reason.
>
> If you want accurate access logging and request information,
> https://twistedmatrix.com/trac/ticket/7704 will probably also be of
> interest to you.
>
> I'm so sorry you've hit this glaring deficiency in Twisted.
>
> On the other hand: I'm so glad that you've hit this glaring deficiency in
> Twisted!  I hope you will be motivated to fix it :-).  It's bothered me for
> quite some time that we don't play nicely with proxying setups, when such
> setups are so *incredibly* common.  If you can write pull requests to fix
> these issues and put them into review, I'm pretty sure you will find an
> enthusiastic reviewer quickly.
>
> -glyph
> _______________________________________________
> Twisted-web mailing list
> Twisted-web at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-web/attachments/20170314/4641784d/attachment.html>