getHostByName Error when DNS Server return Name Server

[wangminghua]

Python Version:2.5 Twisted Version:2.5 Names Version:0.4 There is a bug in the extractRecord function of common.py.

When we got a resolver and call getHostByname, perhaps DNS server can return CNAME, Auth NS Server Name and Auth NS Server IP Address. So extractRecord function will extract the Auth NS Server and send the DNS request to the Auth NS Server. But there is a bug in the extractRecord function.

    # No answers, but maybe there's a hint at who we should be asking about this
    for r in answers:
        if r.type == dns.NS:
            from twisted.names import client
            r = client.Resolver(servers=[(str(r.payload.name), dns.PORT)])
            return r.lookupAddress(str(name)

Generally, the Auth NS Server is the Domain name, and it's IP contained in the addition fields. If we get the NS Server by r.payload.name, we will get the DNS Server Domain, so the Domain as server will pass to client.queryUDP as DNS Server Address. UDP.write will complaint that warnings.warn("Please only pass IPs to write(), not hostnames "+addr[0], DeprecationWarning, stacklevel=2). In addition, self.socket.sendto(datagram, addr) will call socket.gethostbyname(addr), but this call is block.

I think we should use the Auth NS Server IP to replace the Domain. The patch code as fellow:

    # No answers, but maybe there's a hint at who we should be asking about this
    for r in answers:
        if r.type == dns.NS:
            from twisted.names import client
            server = str(r.payload.name)
            
            for s in answers:
                if s.type==dns.A and str(s.name)==nsServerName:
                    server =  socket.inet_ntop(socket.AF_INET, s.payload.address)
                    break

            r = client.Resolver(servers=[(server, dns.PORT)])

[wangminghua]

TEST getHostByName Snippet

[jerub]

wangminghua's code from the ticket description transcribed into a patch

[jerub]

We have no tests for extractRecord and we really should have a unit test for every branch the logic can take.

[wangminghua]

I spell the word of "server" as "nsServerName" when I committed, Sorry , the right patch should as fellow:

# No answers, but maybe there's a hint at who we should be asking about this for r in answers:

if r.type == dns.NS:

from twisted.names import client server = str(r.payload.name)

for s in answers:

if s.type==dns.A and str(s.name)==server :

server = socket.inet_ntop(socket.AF_INET, s.payload.address) break

r = client.Resolver(servers=[(server, dns.PORT)])

[exarkun]

fixing markup in description

[glyph]

#3522 was a duplicate of this.

[Mekk]

Just faced the same (or similar) issue. Moreover the patch does not seem to resolve it. I will attach the testing snippet, let me just mention that twisted resolver fails to resolve info.onet.pl (one of the entry pages of big Polish portal) and www.nozbe.com (reasonably known GTD tool), at least while working from where I am.

I tried the following short script:

from twisted.internet import reactor, defer
import twisted.names.client as dns
import traceback

DOMAIN_NAMES = ('ebay.de', 'google.de', 'info.onet.pl', 'www.nozbe.com', 'nozbe.com')

@defer.inlineCallbacks
def lookup_domain(domain):
    try:
        print "Resolving %s" % domain
        reply = yield dns.getHostByName(domain, timeout = (2,3))
        print "%s ==> %s" % (domain, reply)
    except Exception, e:
        print "%s failed to resolve: %s" % (domain, e)
        traceback.print_exc()

def finished(result):
    reactor.stop()

dlist = []
for domain in DOMAIN_NAMES:
    dlist.append(lookup_domain(domain))
dl = defer.DeferredList(dlist)
dl.addBoth(finished)

reactor.run()

The script took about 30 secs to finish (I felt that timeout option I gave to getHostByName should shorten it, but that's another story) and failed to resolve some of well known names. Here is the result:

Resolving ebay.de
Resolving google.de
Resolving info.onet.pl
Resolving www.nozbe.com
Resolving nozbe.com
/usr/lib/python2.5/site-packages/twisted/names/dns.py:1478: DeprecationWarning: Please only pass IPs to write(), not hostnames
  self.transport.write(message.toStr(), address)
nozbe.com ==> 207.58.151.91
ebay.de ==> 66.135.221.11
google.de failed to resolve: [Query('google.de', 1, 1)]
Traceback (most recent call last):
  File "dns_lookup.py", line 12, in lookup_domain
    reply = yield dns.getHostByName(domain, timeout = (2,3))
TimeoutError: [Query('google.de', 1, 1)]
info.onet.pl failed to resolve: [Query('f3virt.onet.pl', 1, 1)]
Traceback (most recent call last):
  File "dns_lookup.py", line 12, in lookup_domain
    reply = yield dns.getHostByName(domain, timeout = (2,3))
TimeoutError: [Query('f3virt.onet.pl', 1, 1)]
www.nozbe.com failed to resolve: [Query('nozbe.com', 1, 1)]
Traceback (most recent call last):
  File "dns_lookup.py", line 12, in lookup_domain
    reply = yield dns.getHostByName(domain, timeout = (2,3))
TimeoutError: [Query('nozbe.com', 1, 1)]

Tested on Ubuntu Intrepid python-twisted-names package (= version 8.1.0-1)

[Mekk]

Note also, that I tried using the patch attached here (monkey-patching) and it does not seem to help.

[exarkun]

A capture of the DNS traffic which occurs when trying this program might be useful. It's hard to reproduce DNS experiments, since there's so much that can be different on different networks.

[Mekk]

I will take a look, earlier on let me quote one more thing.

I did the following monkey patching in the script above, just to check what is going on:

from twisted.internet import reactor, defer
import twisted.names.client as dnsclient
import twisted.names.common
from twisted.names.common import socket, dns
import traceback

origExtractRecord = twisted.names.common.extractRecord

def patchedExtractRecord(resolver, name, answers, level = 10):
    print "  Extracting %s level %d" % (name, level)
    print "    scanned list contains:"
    for r in answers:
        if r.type == dns.CNAME:
            print "     ", r.name, "CNAME", r.payload.name
        elif r.type == dns.A:
            print "     ", r.name, "A", socket.inet_ntop(socket.AF_INET, r.payload.address)
    return origExtractRecord(resolver, name, answers, level)

twisted.names.common.extractRecord = patchedExtractRecord


DOMAIN_NAMES = ('ebay.de', 'google.de', 'info.onet.pl', 'www.nozbe.com', 'nozbe.com')

@defer.inlineCallbacks
def lookup_domain(domain):
    try:
        print "Resolving %s" % domain
        reply = yield dnsclient.getHostByName(domain, timeout = (2,3))
        print "%s ==> %s" % (domain, reply)
    except Exception, e:
        print "%s failed to resolve: %s" % (domain, e)
        traceback.print_exc()

def finished(result):
    reactor.stop()

dlist = []
for domain in DOMAIN_NAMES:
        dlist.append(lookup_domain(domain))
dl = defer.DeferredList(dlist)
dl.addBoth(finished)

reactor.run()

here are the results:

Resolving ebay.de
Resolving google.de
Resolving info.onet.pl
Resolving www.nozbe.com
Resolving nozbe.com
  Extracting ebay.de level 10
    scanned list contains:
      sjc-dns2.ebaydns.com A 66.135.207.138
      smf-dns2.ebaydns.com A 66.135.215.5
/usr/lib/python2.5/site-packages/twisted/names/dns.py:1478: DeprecationWarning: Please only pass IPs to write(), not hostnames
  self.transport.write(message.toStr(), address)
  Extracting google.de level 10
    scanned list contains:
      ns1.google.com A 216.239.32.10
      ns2.google.com A 216.239.34.10
      ns3.google.com A 216.239.36.10
      ns4.google.com A 216.239.38.10
  Extracting info.onet.pl level 10
    scanned list contains:
      info.onet.pl CNAME f3virt.onet.pl
  Extracting f3virt.onet.pl level 9
    scanned list contains:
      info.onet.pl CNAME f3virt.onet.pl
info.onet.pl failed to resolve: (-2, 'Name or service not known')
Traceback (most recent call last):
  File "dns_lookup_tracing.py", line 30, in lookup_domain
    reply = yield dnsclient.getHostByName(domain, timeout = (2,3))
gaierror: (-2, 'Name or service not known')
  Extracting www.nozbe.com level 10
    scanned list contains:
      www.nozbe.com CNAME nozbe.com
      ns22.domaincontrol.com A 208.109.255.11
  Extracting nozbe.com level 9
    scanned list contains:
      www.nozbe.com CNAME nozbe.com
      ns22.domaincontrol.com A 208.109.255.11
  Extracting nozbe.com level 10
    scanned list contains:
      nozbe.com A 207.58.151.91
      mail.nozbe.com A 207.58.151.91
      ns22.domaincontrol.com A 208.109.255.11
nozbe.com ==> 207.58.151.91
ebay.de failed to resolve: [Query('ebay.de', 1, 1)]
Traceback (most recent call last):
  File "dns_lookup_tracing.py", line 30, in lookup_domain
    reply = yield dnsclient.getHostByName(domain, timeout = (2,3))
TimeoutError: [Query('ebay.de', 1, 1)]
google.de failed to resolve: [Query('google.de', 1, 1)]
Traceback (most recent call last):
  File "dns_lookup_tracing.py", line 30, in lookup_domain
    reply = yield dnsclient.getHostByName(domain, timeout = (2,3))
TimeoutError: [Query('google.de', 1, 1)]
www.nozbe.com failed to resolve: [Query('nozbe.com', 1, 1)]
Traceback (most recent call last):
  File "dns_lookup_tracing.py", line 30, in lookup_domain
    reply = yield dnsclient.getHostByName(domain, timeout = (2,3))
TimeoutError: [Query('nozbe.com', 1, 1)]

I have no clue what is going on but if this routine is called for every candidate list, then it looks like this list is incomplete...

[Mekk]

Here is strace of network traffic generated by my script 192.168.1.1 is the internal DNS used here

[Mekk]

I also made wireshark recording of traffic around port 53, would not like to publish it on the web as it captured also a little bit of other traffic, but I can send it by private email.

[Mekk]

I also made wireshark recording of traffic around port 53, would not like to publish it on the web as it captured also a little bit of other traffic, but I can send it by private email.

(posted to exarkun email at twistedmatrix.com)