Ticket #3014: ipv6.log

17:32 < exarkun> So, I think the thing that makes this difficult is just that Twisted is bad.
17:32 < exarkun> Er I mean listenTCP is a bogus API.
17:33 < exarkun> Hm
17:33 < exarkun> No, I really mean that listenTCP6 is a bogus API.
17:33 < exarkun> I didn't actually look at how that is interacting with name resolution.
17:33 < exarkun> I assume it is somehow though.
17:34 < therve> afaik, the problem is more connectTCP6
17:34 < exarkun> therve: It's the same, although it's more obvious with connectTCP6, yea
17:34 < exarkun> IPv4Address(...).connectStream(...) is a good API
17:34 < exarkun> Because it lets you do IPv6Address(...).connectStream(...) too
17:34 < exarkun> Polymorphism etc!
17:34 < therve> yeah, but it's doesn't exist
17:34 < exarkun> That's true.
17:35 < exarkun> A slightly less good API is
17:35 < therve> making that depend on a ticket opened 3 years ago is unfortunate
17:35 < exarkun> connectTCP(IPv4Address(...), ...)
17:35 < exarkun> since it lets you do connectTCP(IPv6Address(...), ...)
17:35 < exarkun> (Or connectTCP(UNIXAddress(...), ...))
17:35 < radix> or Hostname(...).toIPv6Address()
17:35 < glyph> :-(
17:36 < exarkun> radix: Really?
17:36 < radix> exarkun: probably not. The weird thing about it is that it would return a Deferred.
17:36 < radix> and you probably don't want a connect method that takes a deferred.
17:36 < exarkun> Well!  That's interesting.
17:36 < glyph> is an IPv6 socket a fundamentally different thing from an IPv4 socket?
17:36 < exarkun> You could argue that connectTCP('hostname', ...) is insane.
17:36 < radix> So it'd be Hostname(...).toIPv6Address().addCallback(connectTCP, ...)
17:36 < exarkun> glyph: Those words are missing semantics, I think.
17:37 < glyph> exarkun: I am missing the semantics in my brain why is why I didn't put them in the words.
17:37 < exarkun> Or I could say "IPv6 socket is a nonsensical concept, don't say that ever again"
17:37 < exarkun> The underlying BSD socket API makes it really clear what the high-level abstraction should be
17:37 < glyph> what's the underlying API?
17:38 < exarkun> socket(domain, type, protocol)
17:38 < glyph> oh, that API.
17:38 < radix> what's the clear thing? :)
17:39 < exarkun> The clear thing is that there's a domain and a type (and we'll ignore protocol!  because who cares)
17:39 < exarkun> So the high-level abstraction should have a domain and a type
17:39 < radix> ok, I always forget which of those things you're supposed to ignore :)
17:39 < exarkun> connectTCP makes it hard to support IPv6 because it smashes domain and type into one thing
17:40 < exarkun> connectTCP in Twisted right now is really connectTCPIPv4, and if you want IPv6 you need connectTCPIPv6
17:40 < exarkun> and that's ridiculous
17:40 < exarkun> and you don't even have to look at ipv6 to see how it's ridiculous
17:40 < exarkun> because connectUNIX is the same crummy thing
17:40 < exarkun> So the high-level abstraction should let you talk about domains separately from types
17:41 < exarkun> and it doesn't do that in a good way right now
17:41 < radix> exarkun: maybe connectTCP is still a nice convenience API to have, especially if it will either use IPv6 or IPv4, automatically.
17:41 < exarkun> sure, that's the connectTCP({IPv4,IPv6,UNIX}Address(...), ...) case
17:41 < glyph> right, how do you *tell* if you want to use IPv6 or IPv4 though
17:41 < exarkun> Yes, another good question!
17:41 < radix> exarkun: no, I mean connectTCP(string, ...)
17:42 < exarkun> radix: whatever, same difference, hang on a sec
17:42 < dash> glyph: simple - you don't ever want ipv6
17:42 < exarkun> if you have a "string", then you're thinking about the same thing glyph is thinking about
17:42 < radix> I will hang on, but it doesn't sound like the same thing :)
17:42  * dash runs away
17:42 < exarkun> except again you don't have to bring ipv6 into it to get the problem
17:43 < exarkun> if you pass "example.org" to connectTCP and there's two A records associated with example.org, which do you use? 
17:43 < radix> holy shit, is it raining?
17:43 < exarkun> the way it works now, you make an arbitrary decision, which is a bad thing for the lowest-level api to do.
17:43 < radix> worse! it's deathing
17:43 < radix> exarkun: yeah, definitely.
17:44 < exarkun> so is it any worse if you make the arbitrary decision between two A records, an A6 record, and three AAAA records?
17:44 < radix> exarkun: I was kind of proposing connectTCP becoming a higher level API, while maintaining the same name and signature.
17:44 < glyph> radix: it's *deathing*?
17:44 < exarkun> radix: that might be a good idea.  in any case, disallowing a string to connectTCP now is probably not a good idea.
17:44 < radix> glyph: sleet
17:44 < radix> exarkun: yeah :)
17:45 < radix> exarkun: I guess the thing I'm trying to figure out is whether we should also have it look up an A6 record for that string in the future
17:45 < exarkun> Well
17:45 < radix> (or AAAA or whatever TF we're supposed to use)
17:45 < exarkun> Turns out
17:45 < exarkun> This is pretty funny really
17:45 < radix> heh
17:45 < glyph> oh no
17:45 < radix> are you going to paste that thng
17:46 < exarkun> Turns out it has been looking up A6 and AAAA records for years
17:46 < radix> blirk
17:46 < exarkun> Good thing nobody uses IPv6 or that might've caused some problems.
17:47 < radix> so basically Twisted is really slow :)
17:47 < exarkun> Nah, it's actually no slower than if it only looked up A records!
17:47 < exarkun> However, it is broken (since ANY queries aren't really very good)
17:47 < radix> exarkun: did you learn some things about DNS
17:47 < radix> exarkun: oh wait, the ANY thing??????
17:47 < exarkun> I learned some things about DNS
17:48 < radix> exarkun: SURELY we fixed that
17:48 < radix> I discovered the ANY bug like ten *years* ago
17:48 < exarkun> It may be that no one ever uses anything but the default resolver
17:48 < exarkun> And that's what lets it actually work, I'm not sure
17:48 < radix> ah, ok
17:48 < radix> exarkun: so, yeah, maybe you don't remember
17:48 < radix> exarkun: when I worked for April, I discovered that ANY is something you should never use
17:48 < exarkun> Indeed.
17:48 < exarkun> But apparently you didn't fix it in Twisted.
17:48 < radix> because it often only returns cached values
17:49 < radix> exarkun: crap. did I even file a bug? :(
17:49 < radix> it's about getHostByName, right?
17:49 < exarkun> It's about getHostByName, yes.
17:49 < exarkun> I don't remember any ticket related to this.
17:49 < radix> yeah. it does an ANY query. dangit.
17:49 < radix> I'm sorry. :(
17:49 < exarkun> We're wandering from the main point though, I think
17:50 < exarkun> which is that there should be an api which parameterizes the domain separately from the type
17:50 < exarkun> instead of encoding the type into the method name
17:51 < PenguinOfDoom> IReactorInternetInternetInternetInternet
17:52 < exarkun> therve: I don't have any idea if that answered any of the interesting questions.
17:54 < therve> exarkun: I think for now it mainly raised other questions :/
17:55 < PenguinOfDoom> therve: exarkun is a scientist, not a preacher. That's how all of his explanations are.
17:55 < glyph> exarkun: Well, at some point, the correct answer here is "endpoints", right?
17:55 < therve> glyph: in your dreams :)
17:55 < therve> glyph: this is not only an API issue, unfortunately
17:55 < glyph> therve: Twisted *IS* my dream ;-)
17:55 < PenguinOfDoom> glyph: Twisted isn't pretend vaporware like endpoints, though.
17:56 < PenguinOfDoom> well, mostly
17:56 < glyph> OK, so, I think I'm getting lost on the details that are not API issues
17:56 < glyph> (or bugs in listenTCP)
17:57 < radix> hmm apparently it is called "*" in the RFC, not ANY
17:57 < exarkun> indeed
17:57 < exarkun> but who cares about the rfc, dig says it's "any"
17:58 < radix> exarkun: well yes, I'm just trying to refer to something in an RFC in this ticket I'm filing
17:58 < exarkun> radix: ah, excellent
17:58 < radix> but it is really hard to find the stuff I need
17:58 < exarkun> therve: Since I haven't really looked at the patch in depth, I can't think of any of the non-API issues
17:59 < therve> the great thing is that I only spotted that because for some reason os X decides to return IPV6 records first
17:59 < glyph> therve: sweeeeet
17:59 < exarkun> radix: I can give you a link to some mailing list correspondence to link to from the ticket :)
17:59 < glyph> therve: OS X is clearly the OS of the *future*
17:59 < PenguinOfDoom> therve: That's because the RFC tells to return IPv6 records first.
17:59 < glyph> because in the future the internet will burn down and nothing will work
17:59 < glyph> o/`
18:00 < exarkun> therve: Oh right, getaddrinfo?
18:00 < therve> exarkun: yeah
18:00 < exarkun> That part of the change seemed dubious to me.  But I'm not sure.
18:00 < PenguinOfDoom> glyph: Whatever. In the future, I will be surfing web6 while you are wandering among the ruins of your old crappy internet
18:00 < glyph> PenguinOfDoom: you aren't even surfing web 2 yet
18:00 < radix> I thought I once read something by djb about */ANY, but I can't find it. The only thing I see that he mentions is about how MX records can be interleaved with other records, and that happens especially often in * queries.
18:00 < glyph> no way you're going to get all the way to 6
18:00 < PenguinOfDoom> glyph: Cisco ain't done making the surfboard yet
18:00 < therve> exarkun: maybe
18:01 < PenguinOfDoom> djb hates the hell out of ipv6
18:01 < exarkun> therve: Isn't a simple solution to not use getaddrinfo for IPv4 connection attempts?
18:01 < therve> radix: http://cr.yp.to/djbdns/killa6.html
18:01 < therve> exarkun: well, how would you know?
18:01 < radix> therve: this isn't about the ipv6 stuff, just about what qtype=* means
18:02 < exarkun> therve: connectTCP('a.b.c.d', ...) is an ipv4 connection attempt; connectTCP('a::b', ...) is an ipv6 connection attempt; connectTCP('hostname', ...) is an ipv4 connection attempt
18:02 < exarkun> therve: for example
18:02 < therve> ah
18:02 < therve> that's a solution, indeed
18:02 < glyph> connectTCP(_6=True)
18:02 < therve> glyph: dieinfirenow
18:03 < exarkun> +1 :)
18:03 < glyph> more seriously, presumably someone might want connectTCP('hostname', ...) to be IPv6.  How do they say that?
18:03 < therve> using endpoints! ahah!
18:03 < exarkun> Yes.
18:03 < exarkun> Or more generally, "using the good API instead of the bad API"
18:03 < glyph> I guess that's where I get a little lost.  What would the endpoint call, if not connectTCP6 or listenTCP6?
18:03 < exarkun> maybe that's endpoints maybe it's something less drastically different
18:04 < glyph> I think of endpoints as a higher level API around lower level stuff the reactor is doing.
18:04 < glyph> not as a replacement for all of the existing reactor APIs
18:04 < exarkun> yes.
18:04 < exarkun> you can't do it if it's not implemented by the reactor
18:04 < PenguinOfDoom> no :(
18:04 < exarkun> but the way the reactor wants to work now is not awesome.
18:04 < PenguinOfDoom> endpoints.connectTCP6('ip6porn.com', reactor=ip6reactor)
18:05 < exarkun> I would suggest connectTCP(IPv6Address(...), ...) as a better low-level API than connectTCP6(...)
18:05 < PenguinOfDoom> wait, that doesn't even begin to make sense :(
18:05 < exarkun> PenguinOfDoom: Serious contributions only, please.
18:05 < exarkun> PenguinOfDoom: We will have joke time afterwards.
18:05 < PenguinOfDoom> :(
18:05 < PenguinOfDoom> Okay, do I bring SCTP up now or during joke time?
18:05 < exarkun> PenguinOfDoom: joke time
18:05 < PenguinOfDoom> :(
18:06 < exarkun> We have an example of a different type already - UDP.
18:06 < dreid> PenguinOfDoom: that's not how endpoints are supposed to work.
18:06 < kenaan> new names defect #3019 by radix: twisted.name's default getHostByName does not reliably fetch records because of its use of ANY (qtype=*)
18:06 < glyph> radix: careful with that apostrophe, you might hurt somebody.
18:07 < glyph> exarkun: I think I must have a broken mental model of how this is all supposed to work.  In my fantasy world, a user cares mostly about the hostname, with IPv6 or IPv4 being an irrelevant detail; connectTCP("hostname", ...) should therefore connect to either v6 or v4 hosts as appropriate and not tell anyone.
18:07 < radix> my apostrophes are awesome and deadly.
18:07 < glyph> exarkun: once you've done name resolution by yourself, connectTCP(IPv6Address) seems like a good API.
18:07 < dreid> I agree.
18:07 < exarkun> glyph: HostnameEndpoint("hostname").connectStream(...)
18:07 < radix> hmm, ctrl-t doesn't work in firefox
18:07 < PenguinOfDoom> glyph: In the real world, google.com gets an A6 record and half their users suddenly can't reach the site because they have a link-local ipv6 address, but no real ipv6 connectivity
18:08 < glyph> exarkun: I'm pretty sure I'm confused though.  Perhaps that sort of magic should be reserved for something like strports?
18:08 < radix> glyph: I changed it
18:08 < radix> glyph: I hope you like what I changed it to
18:08 < glyph> PenguinOfDoom: I was beginning to guess that might be the case, yes.
18:08 < PenguinOfDoom> but wait, there's more. Vista *does* have ipv6 connectivity out of the box
18:08 < PenguinOfDoom> so only people behind corporate firewalls will be screwed
18:08 < glyph> PenguinOfDoom: ... but it's all routed through spyware.microsoft.com?
18:09 < PenguinOfDoom> glyph: No, it uses the magical anycast tunnel, I think
18:09 < PenguinOfDoom> not sure how that works with NAT
18:09 < therve> oh yeah, XP ipv6 support seems to be totally broken
18:09 < radix> PenguinOfDoom: whoah, are you serious? 
18:09 < exarkun> PenguinOfDoom: It's easy to recognize a link-local ipv6 address and decide to never try to make ipv6 connections.
18:10 < exarkun> glyph: Did I answer your question or not?
18:10 < PenguinOfDoom> exarkun: Maybe in your fantasy world of well-written C libraries.
18:10 < glyph> exarkun: HostnameEndpoint(...)?
18:10 < exarkun> glyph: Yes.
18:10 < exarkun> PenguinOfDoom: My fantasy world has Python in it, not C.
18:10 < PenguinOfDoom> radix: I tried it once, I think.
18:10 < radix> PenguinOfDoom: did you connect to freenode? :)
18:11 < glyph> exarkun: billions of Twisted programs everywhere are using connectTCP when that's what they mean, but I guess it's not really an important consideration given that we wouldn't be breaking them just by not changing its behavior
18:11 < PenguinOfDoom> exarkun: Python can't even find out what local addresses the box has
18:11 < PenguinOfDoom> radix: Now that gives me an idea.
18:11 < exarkun> glyph: Not changing behavior is awesome.
18:12 < PenguinOfDoom> ah crap, this is where amiaaaaornot.net would come in handy
18:12 < glyph> exarkun: I would prefer to change behavior so that cool new functionality would magically start working with new versions of Twisted, but I can't really get worked up about this particular feature.  IPv6 just makes every particle of my body become sad.
18:12 < PenguinOfDoom> I guess, since dreid is being an ass, I should just register the .org or the .com variant
18:12 < dreid> PenguinOfDoom: I'm almost certain you already control the DNS.
18:12 < exarkun> glyph: Just think about it this way
18:13 < exarkun> glyph: There _is_ no cool new functionality.
18:13 < PenguinOfDoom> oh whoops
18:13 < PenguinOfDoom> dreid: Yes. It's hax time.
18:13 < exarkun> glyph: No one cares about or benefits from IPv6.
18:13 < glyph> exarkun: Exactly what I meant, yes. :)
18:14 < exarkun> Freenode's IPv6 configuration is actually a good example of what the real world constrains you t o.
18:14 < exarkun> There is no IPv6 address for chat.freenode.net.
18:14 < exarkun> If you want ipv6, then you connect to ipv6.chat.freenode.net.
18:14 < glyph> ooh, which ping thinks is an unknown host.
18:15 < glyph> is there a ping6?
18:15 < PenguinOfDoom> ping6, dude
18:15 < glyph> oh god there is
18:15 < glyph> fuck
18:15 < PenguinOfDoom> hahahaha
18:15 < glyph> that was a joke
18:15 < exarkun> and traceroute6 and so on and so forth.
18:15 < glyph> okay
18:15 < PenguinOfDoom> mtr is automagic, though
18:15 < PenguinOfDoom> so's telnet
18:15 < glyph> well why didn't you guys say so before then
18:15 < PenguinOfDoom> -4 or -6
18:15 < glyph> the solution is obvious
18:15 < glyph> we'll only have connectTCP
18:15 < itamar> twisted6
18:15 < itamar> or connectTCP6?
18:15 < glyph> but we'll do sys.argv[0].endswith('6')
18:15 < PenguinOfDoom> :/
18:15 < exarkun> alright, we're in joke time now I guess
18:16 < glyph> twistd6, trial6, etc
18:16 < exarkun> so I'm gonna go get lunch