Opened 7 years ago

Last modified 6 years ago

#2850 defect new

getHostByName Error when DNS Server return Name Server

Reported by: wangminghua Owned by:
Priority: normal Milestone:
Component: names Keywords: getHostByName Block
Cc: Branch:
Author: Launchpad Bug:

Description (last modified by exarkun)

Python Version:2.5
Twisted Version:2.5
Names Version:0.4
There is a bug in the extractRecord function of common.py.

When we got a resolver and call getHostByname, perhaps DNS server can return CNAME, Auth NS Server Name and Auth NS Server IP Address. So extractRecord function will extract the Auth NS Server and send the DNS request to the Auth NS Server. But there is a bug in the extractRecord function.

    # No answers, but maybe there's a hint at who we should be asking about this
    for r in answers:
        if r.type == dns.NS:
            from twisted.names import client
            r = client.Resolver(servers=[(str(r.payload.name), dns.PORT)])
            return r.lookupAddress(str(name)

Generally, the Auth NS Server is the Domain name, and it's IP contained in the addition fields. If we get the NS Server by r.payload.name, we will get the DNS Server Domain, so the Domain as server will pass to client.queryUDP as DNS Server Address. UDP.write will complaint that warnings.warn("Please only pass IPs to write(), not hostnames "+addr[0], DeprecationWarning, stacklevel=2). In addition, self.socket.sendto(datagram, addr) will call socket.gethostbyname(addr), but this call is block.

I think we should use the Auth NS Server IP to replace the Domain.
The patch code as fellow:

    # No answers, but maybe there's a hint at who we should be asking about this
    for r in answers:
        if r.type == dns.NS:
            from twisted.names import client
            server = str(r.payload.name)
            
            for s in answers:
                if s.type==dns.A and str(s.name)==nsServerName:
                    server =  socket.inet_ntop(socket.AF_INET, s.payload.address)
                    break

            r = client.Resolver(servers=[(server, dns.PORT)])

Attachments (3)

test_snippet.py (226 bytes) - added by wangminghua 7 years ago.
TEST getHostByName Snippet
extractRecord.patch (912 bytes) - added by jerub 7 years ago.
wangminghua's code from the ticket description transcribed into a patch
strace.txt (26.0 KB) - added by Mekk 6 years ago.
Here is strace of network traffic generated by my script 192.168.1.1 is the internal DNS used here

Download all attachments as: .zip

Change History (14)

Changed 7 years ago by wangminghua

TEST getHostByName Snippet

Changed 7 years ago by jerub

wangminghua's code from the ticket description transcribed into a patch

comment:1 Changed 7 years ago by jerub

We have no tests for extractRecord and we really should have a unit test for every branch the logic can take.

comment:2 Changed 7 years ago by wangminghua

I spell the word of "server" as "nsServerName" when I committed, Sorry , the right patch should as fellow:

# No answers, but maybe there's a hint at who we should be asking about this for r in answers:

if r.type == dns.NS:

from twisted.names import client
server = str(r.payload.name)

for s in answers:

if s.type==dns.A and str(s.name)==server :

server = socket.inet_ntop(socket.AF_INET, s.payload.address)
break

r = client.Resolver(servers=[(server, dns.PORT)])

comment:3 Changed 7 years ago by exarkun

  • Description modified (diff)
  • Milestone Names-0.4 deleted

fixing markup in description

comment:4 Changed 6 years ago by glyph

#3522 was a duplicate of this.

comment:5 Changed 6 years ago by Mekk

Just faced the same (or similar) issue. Moreover the patch does not seem to resolve it.
I will attach the testing snippet, let me just mention that twisted resolver fails to resolve info.onet.pl (one of the entry pages of big Polish portal) and www.nozbe.com (reasonably known GTD tool), at least while working from where I am.

I tried the following short script:

from twisted.internet import reactor, defer
import twisted.names.client as dns
import traceback

DOMAIN_NAMES = ('ebay.de', 'google.de', 'info.onet.pl', 'www.nozbe.com', 'nozbe.com')

@defer.inlineCallbacks
def lookup_domain(domain):
    try:
        print "Resolving %s" % domain
        reply = yield dns.getHostByName(domain, timeout = (2,3))
        print "%s ==> %s" % (domain, reply)
    except Exception, e:
        print "%s failed to resolve: %s" % (domain, e)
        traceback.print_exc()

def finished(result):
    reactor.stop()

dlist = []
for domain in DOMAIN_NAMES:
    dlist.append(lookup_domain(domain))
dl = defer.DeferredList(dlist)
dl.addBoth(finished)

reactor.run()

The script took about 30 secs to finish (I felt that timeout option I gave to getHostByName should shorten it, but that's another story) and failed to resolve some of well known names. Here is the result:

Resolving ebay.de
Resolving google.de
Resolving info.onet.pl
Resolving www.nozbe.com
Resolving nozbe.com
/usr/lib/python2.5/site-packages/twisted/names/dns.py:1478: DeprecationWarning: Please only pass IPs to write(), not hostnames
  self.transport.write(message.toStr(), address)
nozbe.com ==> 207.58.151.91
ebay.de ==> 66.135.221.11
google.de failed to resolve: [Query('google.de', 1, 1)]
Traceback (most recent call last):
  File "dns_lookup.py", line 12, in lookup_domain
    reply = yield dns.getHostByName(domain, timeout = (2,3))
TimeoutError: [Query('google.de', 1, 1)]
info.onet.pl failed to resolve: [Query('f3virt.onet.pl', 1, 1)]
Traceback (most recent call last):
  File "dns_lookup.py", line 12, in lookup_domain
    reply = yield dns.getHostByName(domain, timeout = (2,3))
TimeoutError: [Query('f3virt.onet.pl', 1, 1)]
www.nozbe.com failed to resolve: [Query('nozbe.com', 1, 1)]
Traceback (most recent call last):
  File "dns_lookup.py", line 12, in lookup_domain
    reply = yield dns.getHostByName(domain, timeout = (2,3))
TimeoutError: [Query('nozbe.com', 1, 1)]

Tested on Ubuntu Intrepid python-twisted-names package (= version 8.1.0-1)

comment:6 Changed 6 years ago by Mekk

Note also, that I tried using the patch attached here (monkey-patching) and it does not seem to help.

comment:7 Changed 6 years ago by exarkun

A capture of the DNS traffic which occurs when trying this program might be useful. It's hard to reproduce DNS experiments, since there's so much that can be different on different networks.

comment:8 Changed 6 years ago by Mekk

I will take a look, earlier on let me quote one more thing.

I did the following monkey patching in the script above, just to check what is going on:

from twisted.internet import reactor, defer
import twisted.names.client as dnsclient
import twisted.names.common
from twisted.names.common import socket, dns
import traceback

origExtractRecord = twisted.names.common.extractRecord

def patchedExtractRecord(resolver, name, answers, level = 10):
    print "  Extracting %s level %d" % (name, level)
    print "    scanned list contains:"
    for r in answers:
        if r.type == dns.CNAME:
            print "     ", r.name, "CNAME", r.payload.name
        elif r.type == dns.A:
            print "     ", r.name, "A", socket.inet_ntop(socket.AF_INET, r.payload.address)
    return origExtractRecord(resolver, name, answers, level)

twisted.names.common.extractRecord = patchedExtractRecord


DOMAIN_NAMES = ('ebay.de', 'google.de', 'info.onet.pl', 'www.nozbe.com', 'nozbe.com')

@defer.inlineCallbacks
def lookup_domain(domain):
    try:
        print "Resolving %s" % domain
        reply = yield dnsclient.getHostByName(domain, timeout = (2,3))
        print "%s ==> %s" % (domain, reply)
    except Exception, e:
        print "%s failed to resolve: %s" % (domain, e)
        traceback.print_exc()

def finished(result):
    reactor.stop()

dlist = []
for domain in DOMAIN_NAMES:
        dlist.append(lookup_domain(domain))
dl = defer.DeferredList(dlist)
dl.addBoth(finished)

reactor.run()

here are the results:

Resolving ebay.de
Resolving google.de
Resolving info.onet.pl
Resolving www.nozbe.com
Resolving nozbe.com
  Extracting ebay.de level 10
    scanned list contains:
      sjc-dns2.ebaydns.com A 66.135.207.138
      smf-dns2.ebaydns.com A 66.135.215.5
/usr/lib/python2.5/site-packages/twisted/names/dns.py:1478: DeprecationWarning: Please only pass IPs to write(), not hostnames
  self.transport.write(message.toStr(), address)
  Extracting google.de level 10
    scanned list contains:
      ns1.google.com A 216.239.32.10
      ns2.google.com A 216.239.34.10
      ns3.google.com A 216.239.36.10
      ns4.google.com A 216.239.38.10
  Extracting info.onet.pl level 10
    scanned list contains:
      info.onet.pl CNAME f3virt.onet.pl
  Extracting f3virt.onet.pl level 9
    scanned list contains:
      info.onet.pl CNAME f3virt.onet.pl
info.onet.pl failed to resolve: (-2, 'Name or service not known')
Traceback (most recent call last):
  File "dns_lookup_tracing.py", line 30, in lookup_domain
    reply = yield dnsclient.getHostByName(domain, timeout = (2,3))
gaierror: (-2, 'Name or service not known')
  Extracting www.nozbe.com level 10
    scanned list contains:
      www.nozbe.com CNAME nozbe.com
      ns22.domaincontrol.com A 208.109.255.11
  Extracting nozbe.com level 9
    scanned list contains:
      www.nozbe.com CNAME nozbe.com
      ns22.domaincontrol.com A 208.109.255.11
  Extracting nozbe.com level 10
    scanned list contains:
      nozbe.com A 207.58.151.91
      mail.nozbe.com A 207.58.151.91
      ns22.domaincontrol.com A 208.109.255.11
nozbe.com ==> 207.58.151.91
ebay.de failed to resolve: [Query('ebay.de', 1, 1)]
Traceback (most recent call last):
  File "dns_lookup_tracing.py", line 30, in lookup_domain
    reply = yield dnsclient.getHostByName(domain, timeout = (2,3))
TimeoutError: [Query('ebay.de', 1, 1)]
google.de failed to resolve: [Query('google.de', 1, 1)]
Traceback (most recent call last):
  File "dns_lookup_tracing.py", line 30, in lookup_domain
    reply = yield dnsclient.getHostByName(domain, timeout = (2,3))
TimeoutError: [Query('google.de', 1, 1)]
www.nozbe.com failed to resolve: [Query('nozbe.com', 1, 1)]
Traceback (most recent call last):
  File "dns_lookup_tracing.py", line 30, in lookup_domain
    reply = yield dnsclient.getHostByName(domain, timeout = (2,3))
TimeoutError: [Query('nozbe.com', 1, 1)]

I have no clue what is going on but if this routine is called for every candidate list, then it looks like this list is incomplete...

Changed 6 years ago by Mekk

Here is strace of network traffic generated by my script 192.168.1.1 is the internal DNS used here

comment:9 follow-up: Changed 6 years ago by Mekk

I also made wireshark recording of traffic around port 53, would not like to publish it on the web as it captured also a little bit of other traffic, but I can send it by private email.

comment:10 in reply to: ↑ 9 Changed 6 years ago by Mekk

I also made wireshark recording of traffic around port 53, would not like to publish it on the web as it captured also a little bit of other traffic, but I can send it by private email.

(posted to exarkun email at twistedmatrix.com)

comment:11 Changed 4 years ago by <automation>

  • Owner exarkun deleted
Note: See TracTickets for help on using tickets.