Version 15 (modified by Richard Wall, 7 years ago) (diff)

Removed various junk from the intro. Tidied up the intro. Added much more complete milestones for the EDNS part of the project.



Twisted includes a comprehensive set of DNS components, collectively known as twisted Names.

This is a plan for the development of EDNS(0) and DNSSEC support in Twisted Names.

High level goals

  • New public APIs will be as narrow as possible, to minimise future backwards compatibility complications.
  • Zero changes to existing public APIs. (if possible)
  • Existing classes will be wrapped where appropriate and in other cases, shared functionality will be extracted and re-factored for reuse in new classes.
  • Old sub-optimal APIs will be deprecated and eventually removed after the introduction new EDNS and DNSSEC APIs.

Existing Implementations

Re-use of existing Python DNS libraries (or C libraries with Python bindings) will be considered according to the following criteria:

  1. MIT compatible licence
  2. Actively maintained
  3. Full test coverage

However the potential time savings from reusing an existing library will be weighed against the overhead of introducing an additional Twisted install dependency.

Examples of existing libraries / implementations:

Other sources:


Before we can fully support DNSSEC, we need to first implement EDNS(0) support.

Some DNSSEC APIS (lookup methods, record types) can be implemented in parallel and will be immediately use-able.


Implement EDNS specific record, message, protocol, client and server APIs and switch the twistd dns server plugin to use them.

Deprecate the original non-EDNS aware versions of these.

Here are the steps:

Add Authentic Data (AD) and Checking Disabled (CD) flags

Add two new fields to the existing dns.Message class


  1. Set / get AD and CD field
  2. Byte encode / decode AD and CD field


Parse and construct OPT Pseudo-RR


  1. Get / set fixed OPT record fields.
  2. Byte encode / decode EDNS OPT records.
  3. Convert to / from dns.RRHeader (the existing non-EDNS record header class)
  4. Get / set variable OPT record fields.
  5. Byte encode / decode variable OPT record fields.
  6. Convert to / from dns.UnknownRecord (the existing non-EDNS record payload class for the unknown record payload types.)


Parse and construct EDNS messages


  1. Get / set all non-EDNS message fields
  2. Get / set all new EDNS(0) message fields
  3. Byte encoding / decoding
    1. During encoding, EDNS specific fields will be used to generate a dns.OPTHeader instance which will be added to the end of the additional records list.
    2. During decoding, an OPT pseudo record found in the additional section will be extracted, parsed and its fields will be used to populate the EDNS specific attributes of the EDNSMessage instance.
  4. Get / set extended RCODE
  5. A mechanism for getting protocol errors encountered during parsing. eg
  6. Convert to / from dns.Message (the existing non-EDNS message class)


  1. We will not implement an API for manipulating OPT variable fields at this stage. See "Variable EDNS0 Option Codes" below.

dns.EDNSStreamProtocol and dns.EDNSDatagramProtcol or Message override options

The existing dns.DNSDatagramProtocol and DNSProtocol (TCP) are hardcoded to use dns.Message for decoding and encoding wire messages.

The simplest solution would be to add a new "messageFactory" constructor argument which allows us to supply a curried dns.EDNSMessage instance whose EDNS specific constructor arguments have been preassigned.

Alternatively it might be better to introduce new narrower EDNS Protocol APIs which wrap DNSProtocol, and DNSDatagramProtocol and hide many of their ugly implementation details.

The wrappers can override the "writeMessage" method of the original protocols as demonstrated here:


  1. DNS datagram and stream protocol implementations which can use dns.EDNSMessage for decoding and encoding wire messages.
  2. Get / set EDNS specific options which will be passed to the construct the EDNSMessage instance responsible for encoding and decoding.

client.Resolver protocol override options

client.Resolver is currently hard coded to use dns.DNSDatagramProtocol and dns.DNSProtocol (TCP) via client.DNSClientFactory.

Allow the caller to supply alternative protocol factories.


  1. client.Resolver constructor will accept a "datagramProtocolFactory" and "streamProtocolFactory" arguments, which will default to existing factories but which will allow dns.EDNSDatagramProtocol to be supplied instead.

client.EDNSResolver Fixed UDP Payload Size

Start with a basic EDNSResolver with a fixed UDP payload size and which only does TCP fallback.


  1. client.Resolver constructor will accept a "protocolFactory" argument, which will default to dns.Message but which will allow dns.EDNSMessage (or a factory function) to be supplied instead (break out into separate ticket)


  1. An implementation of t.i.interfaces.IResolver which sends EDNS(0) queries by default.
    • Wraps client.Resolver
    • Supplies an EDNSMessage factory to allow setting EDNS specific fields for all queries.
  2. Get / set AD, CD
  3. Get / set DO field. Default to "unset"
  4. Get / set VERSION. Default to 0.
  5. Get / set maximum UDP payload size.
  6. Detect servers which do not support EDNS(0) and fall back.
    1. ONLY fallback if DO is "unset"
    2. non-EDNS UDP query
    3. TCP query


  1. We will not implement an API for manipulating OPT variable fields at this stage. See "Variable EDNS0 Option Codes" below.

client.EDNSResolver with Automatic Payload Size Selection and Detection

Extend client.EDNSResolver to detect timeouts and fragmentation caused by UDP payload size limits of the server and intermediate devices. Automatically re-issue with successively smaller advertised payload sizes, possibly starting with the server advertised payload size found in the first reply.

Consider postponing this. It's probably not required for basic DNSSEC client support.

"A requestor MAY choose to implement a fallback to smaller advertised sizes to work around firewall or other network limitations."


  1. Issue parallel queries with a small delay between each and with successively smaller advertised UDP payload sizes.
  2. Cache and re-use the detected maximum payload size for each server.
  3. Fallback to TCP query.


  1. "Due to transaction overhead, it is not recommended to advertise an architectural limit as a maximum UDP payload size"
  2. "Values of less than 512 bytes MUST be treated as equal to 512 bytes."


server.DNSServerFactory is responsible for coordinating datagram and stream listening ports and their protocols.

The protocols dispatch their received queries to DNSServerFactory and it in turn dispatches the queries to one or more authoritative, caching or recursive IResolver instances.

It then constructs a response message, populates the answers, authority and additional lists and supplies it to the write method of the appropriate protocol instance.

DNSServerFactory is also responsible for enforcing policy, such as checking the origin of queries or notify messages.

One problem with the existing implementation is that it uses the incoming message and modifies it in place before sending it back to the client. This results in various request fields being returned to the client as if they are the server's chosen response fields.

This is completely wrong for the EDNS payload size and the new AD and CD fields.

A related problem is that it also includes the client's original OPT record in error responses #6645

So here we will implement an EDNSServerFactory somehow overrides the "messageReceived" method in order to enforce correct server max UDP payload size.

Instead of re-using the incoming message instance, we will instead construct a new instance and carefully choose the field values based on the servers configuration.


  1. Get / set maximum UDP payload size.
  2. Send EDNS0 responses to EDNS0 queries
  3. Do not send EDNS0 responses to standard DNS queries.
  4. Respond clients using their advertised UDP payload size. (up to the server maximum).
  5. Limit additional RRSETs so that responses fit the client max UDP payload size or mark response Messages truncated.
  6. Set extended EDNS RCODEs in responses where appropriate.

twistd dns plugin - EDNS(0) by default

At this point we should have enough infrastructure to allow the twistd dns plugin to correctly respond to EDNS(0) queries.

If configured as a forwarding resolver, it will be able to issue EDNS(0) queries to the upstream servers.

For compatibility we may choose to make it respond to clients with non-DNS messages, but using the new EDNSMessage API.

Then at a later date we can add an --edns flag to allow the EDNS features to be turned on or off. Or we could do it all in this ticket.

This isn't strictly needed for the goal of having a DNSSEC validating client, but it will be a nice way in which Twisted users can quickly benefit from the new EDNS APIs.

Variable EDNS0 Option Codes (OPT)

An OPT record payload can contain one or more variable fields. The list of defined fields is here:

One example is the EDNS(0) owner option which allows a DNS proxy (recursive or forwarding resolver) to include information about the origin of the client query (the client subnet), when it queries an authoritative server. An authoritative server can use this information to do DNS based global load balancing.

Each of these variable fields would need it's own API so we need to give thought to how the EDNSResolver can be made extensible enough to support these.

RRSET improvements

DNSSEC seems to rely on stable RRSETs. eg for signatures, for predictable caching of records and signatures, for predictable truncation of large DNSSEC responses.

There appear to be various problems with twisted.names current handling of RRSETs.

Canonical Form and Order of Resource Records

RRSETs must be arranged in canonical order before their signatures are calculated / verified.

This ticket will introduce an algorithm for sorting records according to the rules described in

Sorting appears to be the responsibility of the verifying client not the server.

The fact that Bind has config options for changing the order of RRSETS on the server side and the client side suggests that the canonical ordering of records should be done only for the purpose of DNSSEC validation. It should probably not change the order of records returned by various IResolver methods.

Serving RRSETs

twisted.names.dns.Message should follow the guidance in RFC2181 regarding the handling of TTLs for records in the same RRSET. Alternatively consider handling this higher up in t.n.authority.FileAuthority, t.n.cache, t.n.resolver etc.

Receiving RRSETs

twisted.names.client should somehow signal an error if it receives RRSETs whose RRs have different TTLs.

twisted.names.resolver,root should discard RRSETs whose RRs have diffent TTLs.

  • "Should a client receive a response containing RRs from an RRSet with differing TTLs, it should treat this as an error. If the RRSet concerned is from a non-authoritative source for this data, the client should simply ignore the RRSet,"

Caching RRSETs

twisted.names.cache should follow the RRSET ranking guidance when serving and replacing items in its cache.

  • "Servers must never merge RRs from a response with RRs in their cache to form an RRSet. If a response contains data that would form an RRSet with data in a server's cache the server must either ignore the RRs in the response, or discard the entire RRSet currently in the cache, as appropriate."
  • "When considering whether to accept an RRSet in a reply, or retain an RRSet already in its cache instead, a server should consider the relative likely trustworthiness of the various data"

Selective truncate

twisted.names.dns.Message currently truncates messages based on the combined length of the answer, auth, and additional sections. Instead it (or something higher up) should look at the length of the answers and only set the truncate flag if the entire answers RRSET is greater than the maxPayloadSize. If there is room, the auth and additional records can also be included, but entire RRSETs must be included or none.


Much of this work (including tests) has already been implemented by BobNovas in two large patches attached to original tickets: #5450, #5453, #5454. Look there before implementing anything from scratch.

New DNSSEC Records and Lookup Methods

  1. DNSSEC introduces six new resource record types. Each new record type will require a new dns.Record subclass and a new lookupMethod added to t.i.interfaces.IResolver, t.n.common.ResolverBase and a corresponding free function in t.n.client.
    1. #6664 DNSKey
    2. #6665 RRSIG
    3. NSEC
    4. DS
    5. NSEC3
    6. NSEC3Param
  2. For ease of review, this work can be split into six tickets.
  3. These new records can be implemented independently of EDNS and independently of DNSSEC validation and new DNSSEC related message headers.
  4. Initially, this will allow twisted.names clients to explicitly request these DNSSEC related records.
  5. lookupZone will return DNSSEC related records when transferring from DNSSEC authoritative servers.
  6. t.n.secondary.SecondaryAuthority will download DNSSEC records and serve them when queries specifically ask for them by type.
  7. t.n.authority.FileAuthority will load DNSSEC records and serve them when queries specifically ask for them by type.

Security-aware Non-validating Client

A twisted.names.client will be able to generate EDNS queries with one or both the DO bit and the AD bit set.

It will examine the state of the AD bit in the response to determine whether the upstream resolver claims to have validated the records in the response.

If the DO bit was set, it will expect to receive DNSSEC related records in the response.

Validating Client

A twisted.names.client which sends DO + CD flagged queries and performs its own validation of the returned DNSSEC signatures.

TODO: needs more thought.

Validating Recursive / Forwarding Server

This can use the Validating client API above, but may need to do some processing of answers based on the query flags.

TODO: needs more thought.

DNSSEC Aware Authoritative Server

A twisted.names.authority API which knows where and when to include RRSIG, DNSKEY, NSEC, NSEC3 records etc with responses.

The actual generation of the DNSSEC records can be performed using external tools such as dnssec-signzone

TODO: needs more thought.



  3. 2181: Clarifications to the DNS Specification
  4. 4697: Observed DNS Resolution Misbehavior
  5. 5625: DNS Proxy Implementation Guidelines


  1. 6891: Extension Mechanisms for DNS (EDNS(0))


  1. 4033: DNS Security Introduction and Requirements
  2. 4034: Resource Records for the DNS Security Extensions
  3. 4035: Protocol Modifications for the DNS Security Extensions
  4. 5155: DNS Security (DNSSEC) Hashed Authenticated Denial of Existence
  5. 6781: DNSSEC Operational Practices, Version 2
  6. 6840: Clarifications and Implementation Notes for DNS Security (DNSSEC)
  7. 3833: Threat Analysis of the Domain Name System (DNS)

MISC / TODO / Notes

  1. "If a query message with more than one OPT RR is received, a FORMERR (RCODE=1) MUST be returned." RFC6891 6.1.1
    1. Multiple OPT RR will be detected in EDNSMessage.decode
    2. If multiple OPT RR are detected, set EDNSMessage.decodingErrors.append(EFORMAT)
    3. DNSServerFactory.messageReceived checks message.decodingErrors
    4. If not empty, it sets message.rCode = message.decodingErrors.pop(0) before sending the message back to client without further processing.
    5. If DNSServerFactory.verbose logging is enabled, all message.decodingErrors will be logged.
    6. I tested Google DNS and Bind Authoritative DNS responses to multi OPT queries.
      1. Google seems to ignore all but the first OPT RR and responds without error.
      2. Bind responds with RCODE=1
        } else if (rdtype == dns_rdatatype_opt) {
        * The name of an OPT record must be ".", it
        * must be in the additional data section, and
        * it must be the first OPT we've seen.
                if (!dns_name_equal(dns_rootname, name) ||
                    msg->opt != NULL)
                skip_name_search = ISC_TRUE;
                skip_type_search = ISC_TRUE;
      4. PowerDNS appears to ignore all but the first OPT RR (untested)
        bool getEDNSOpts(const MOADNSParser& mdp, EDNSOpts* eo)
          if(mdp.d_header.arcount && !mdp.d_answers.empty()) {
            BOOST_FOREACH(const MOADNSParser::answers_t::value_type& val, mdp.d_answers) {
              if(val.first.d_place == DNSRecord::Additional && val.first.d_type == QType::OPT) {
                EDNS0Record stuff;
                uint32_t ttl=ntohl(val.first.d_ttl);
                memcpy(&stuff, &ttl, sizeof(stuff));
                eo->d_Z = ntohs(stuff.Z);
                OPTRecordContent* orc =
                  return false;
                return true;
          return false;
  1. "EXTENDED-RCODE Forms the upper 8 bits of extended 12-bit RCODE" RFC6891 6.1.3
    1. Decode: EDNSMessage.decode will merge the RCODE and OPTHeader.extendedRCODE setting the rCode attribute.
    2. Encode: If rCode is > 15, EDNSMessage.encode will set the lower 4 bits as RCODE and set the upper 8 bits on a new or existing OPTHeader which will be added to the additional section.
  1. "VERSION Indicates the implementation level of the setter" RFC6891 6.1.3
    1. "If a responder does not implement the VERSION level of the request, then it MUST respond with RCODE=BADVERS"
      1. Decode: in EDNSMessage.decode, if version > 0, EDNSMessage.decodingErrors.append(EBADVERS)
  1. "This document assigns EDNS Extended RCODE 16 to "BADVERS" in the DNS RCODES registry"
    1. Create dns.EBADVERS = 16
  1. "DO DNSSEC OK bit as defined by [RFC3225]."
    1. Set EDNSMessage.dnssecOK during decode using value from OPTHeader.dnssecOK.
    2. Add or update an existing OPTHeader instance during encode.