wiki:EDNS0

Version 18 (modified by rwall, 9 months ago) (diff)

Milestones / tickets for RRSet improvements. Add various appendices

EDNS Plan

Twisted includes a comprehensive set of DNS components, collectively known as twisted Names.

This is a plan for the development of EDNS(0) and DNSSEC support in Twisted Names.

  • Contributions will be submitted as MIT licensed patches to the Twisted project.
  • New public APIs will be as narrow as possible, to minimise future backwards compatibility complications.
  • Zero changes to existing public APIs. (if possible)
  • Existing classes will be wrapped where appropriate and in other cases, shared functionality will be extracted and re-factored for reuse in new classes.
  • Old sub-optimal APIs will be deprecated and eventually removed after the introduction new EDNS and DNSSEC APIs.

Milestones

Before we can fully support DNSSEC, we need to first implement EDNS(0) support.

Some DNSSEC APIS (lookup methods, record types) can be implemented in parallel and will be immediately use-able.

EDNS(0)

Implement EDNS specific record, message, protocol, client and server APIs and switch the twistd dns server plugin to use them.

Deprecate the original non-EDNS aware versions of these.

Here are the steps:

#6680 Add Authentic Data (AD) and Checking Disabled (CD) flags

Add two new fields to the existing dns.Message class

Requirements:

  1. Set / get AD and CD field
  2. Byte encode / decode AD and CD field

#5668 dns.OPTHeader

Parse and construct OPT Pseudo-RR

Requirements:

  1. Get / set fixed OPT record fields.
  2. Byte encode / decode EDNS OPT records.
  3. Convert to / from dns.RRHeader (the existing non-EDNS record header class)
  4. Get / set variable OPT record fields.
  5. Byte encode / decode variable OPT record fields.
  6. Convert to / from dns.UnknownRecord (the existing non-EDNS record payload class for the unknown record payload types.)

#5675 dns.EDNSMessage

Parse and construct EDNS messages

Requirements:

  1. Get / set all non-EDNS message fields
  2. Get / set all new EDNS(0) message fields
  3. Byte encoding / decoding
    1. During encoding, EDNS specific fields will be used to generate a dns.OPTHeader instance which will be added to the end of the additional records list.
    2. During decoding, an OPT pseudo record found in the additional section will be extracted, parsed and its fields will be used to populate the EDNS specific attributes of the EDNSMessage instance.
  4. Get / set extended RCODE
  5. A mechanism for getting protocol errors encountered during parsing. eg
  6. Convert to / from dns.Message (the existing non-EDNS message class)

Exclusions:

  1. We will not implement an API for manipulating OPT variable fields at this stage. See "Variable EDNS0 Option Codes" below.

dns.EDNSStreamProtocol and dns.EDNSDatagramProtcol or Message override options

The existing dns.DNSDatagramProtocol and DNSProtocol (TCP) are hardcoded to use dns.Message for decoding and encoding wire messages.

The simplest solution would be to add a new "messageFactory" constructor argument which allows us to supply a curried dns.EDNSMessage instance whose EDNS specific constructor arguments have been preassigned.

Alternatively it might be better to introduce new narrower EDNS Protocol APIs which wrap DNSProtocol, and DNSDatagramProtocol and hide many of their ugly implementation details.

The wrappers can override the "writeMessage" method of the original protocols as demonstrated here:

Requirements:

  1. DNS datagram and stream protocol implementations which can use dns.EDNSMessage for decoding and encoding wire messages.
  2. Get / set EDNS specific options which will be passed to the construct the EDNSMessage instance responsible for encoding and decoding.

client.Resolver protocol override options

client.Resolver is currently hard coded to use dns.DNSDatagramProtocol and dns.DNSProtocol (TCP) via client.DNSClientFactory.

Allow the caller to supply alternative protocol factories.

Requirements:

  1. client.Resolver constructor will accept a "datagramProtocolFactory" and "streamProtocolFactory" arguments, which will default to existing factories but which will allow dns.EDNSDatagramProtocol to be supplied instead.

#5670 client.EDNSResolver Fixed UDP Payload Size

Start with a basic EDNSResolver with a fixed UDP payload size and which only does TCP fallback.

Prerequisites:

  1. client.Resolver constructor will accept a "protocolFactory" argument, which will default to dns.Message but which will allow dns.EDNSMessage (or a factory function) to be supplied instead (break out into separate ticket)

Requirements:

  1. An implementation of t.i.interfaces.IResolver which sends EDNS(0) queries by default.
    • Wraps client.Resolver
    • Supplies an EDNSMessage factory to allow setting EDNS specific fields for all queries.
  2. Get / set AD, CD
  3. Get / set DO field. Default to "unset"
  4. Get / set VERSION. Default to 0.
  5. Get / set maximum UDP payload size.
  6. Detect servers which do not support EDNS(0) and fall back.
    1. ONLY fallback if DO is "unset"
    2. non-EDNS UDP query
    3. TCP query

Exclusions:

  1. We will not implement an API for manipulating OPT variable fields at this stage. See "Variable EDNS0 Option Codes" below.

client.EDNSResolver with Automatic Payload Size Selection and Detection

Extend client.EDNSResolver to detect timeouts and fragmentation caused by UDP payload size limits of the server and intermediate devices. Automatically re-issue with successively smaller advertised payload sizes, possibly starting with the server advertised payload size found in the first reply.

Consider postponing this. It's probably not required for basic DNSSEC client support.

"A requestor MAY choose to implement a fallback to smaller advertised sizes to work around firewall or other network limitations."

Requirements:

  1. Issue parallel queries with a small delay between each and with successively smaller advertised UDP payload sizes.
  2. Cache and re-use the detected maximum payload size for each server.
  3. Fallback to TCP query.

Discussion:

  1. "Due to transaction overhead, it is not recommended to advertise an architectural limit as a maximum UDP payload size"
  2. "Values of less than 512 bytes MUST be treated as equal to 512 bytes."

#5669 server.EDNSServerFactory

server.DNSServerFactory is responsible for coordinating datagram and stream listening ports and their protocols.

The protocols dispatch their received queries to DNSServerFactory and it in turn dispatches the queries to one or more authoritative, caching or recursive IResolver instances.

It then constructs a response message, populates the answers, authority and additional lists and supplies it to the write method of the appropriate protocol instance.

DNSServerFactory is also responsible for enforcing policy, such as checking the origin of queries or notify messages.

One problem with the existing implementation is that it uses the incoming message and modifies it in place before sending it back to the client. This results in various request fields being returned to the client as if they are the server's chosen response fields.

This is completely wrong for the EDNS payload size and the new AD and CD fields.

A related problem is that it also includes the client's original OPT record in error responses #6645

So here we will implement an EDNSServerFactory somehow overrides the "messageReceived" method in order to enforce correct server max UDP payload size.

Instead of re-using the incoming message instance, we will instead construct a new instance and carefully choose the field values based on the servers configuration.

Requirements:

  1. Get / set maximum UDP payload size.
  2. Send EDNS0 responses to EDNS0 queries
  3. Do not send EDNS0 responses to standard DNS queries.
  4. Respond clients using their advertised UDP payload size. (up to the server maximum).
  5. Limit additional RRSETs so that responses fit the client max UDP payload size or mark response Messages truncated.
  6. Set extended EDNS RCODEs in responses where appropriate.

twistd dns plugin - EDNS(0) by default

At this point we should have enough infrastructure to allow the twistd dns plugin to correctly respond to EDNS(0) queries.

If configured as a forwarding resolver, it will be able to issue EDNS(0) queries to the upstream servers.

For compatibility we may choose to make it respond to clients with non-DNS messages, but using the new EDNSMessage API.

Then at a later date we can add an --edns flag to allow the EDNS features to be turned on or off. Or we could do it all in this ticket.

This isn't strictly needed for the goal of having a DNSSEC validating client, but it will be a nice way in which Twisted users can quickly benefit from the new EDNS APIs.

Variable EDNS0 Option Codes (OPT)

An OPT record payload can contain one or more variable fields. The list of defined fields is here:

One example is the EDNS(0) owner option which allows a DNS proxy (recursive or forwarding resolver) to include information about the origin of the client query (the client subnet), when it queries an authoritative server. An authoritative server can use this information to do DNS based global load balancing.

Each of these variable fields would need it's own API so we need to give thought to how the EDNSResolver can be made extensible enough to support these.

RRSET improvements

DNSSEC relies on stable RRSETs. eg for signatures, for predictable caching of records and signatures and for predictable truncation of large DNSSEC responses.

RRSETs must be arranged in canonical order before their signatures are calculated / verified.

These tickets will introduce algorithms for sorting records according to the rules described in https://tools.ietf.org/html/rfc4034#section-6.

Sorting appears to be the responsibility of the verifying client not the server.

The fact that Bind has config options for changing the order of RRSETS on the server side and the client side suggests that the canonical ordering of records should be done only for the purpose of DNSSEC validation. It should probably not change the order of records returned by various IResolver methods.

dns.Name.canonical - Canonical form and order of DNS names

Requirements:

  1. A "canonical" property will be added to dns.Name which will return the name in RFC4034 canonical form as bytes. + https://tools.ietf.org/html/rfc4034#section-6.1
  2. Names can be sorted canonically by passing the "canonical" attribute name as the "key" to sorted

dns.IRecord.canonical - Canonical Form and Order of Resource Records

Requirements:

  1. All IRecord implementations will be given a "canonical" method which returns bytes representing the RR in canonical wire format.
  2. A "canonical" method will be added to dns.IRecord
  3. Records can be sorted canonically by passing the "canonical" attribute name as the "key" to sorted

dns.RRSet

"An RRset is not allowed to contain duplicate records (multiple RRs with the same owner name, class, type, and RDATA). Therefore, if an implementation detects duplicate RRs when putting the RRset in canonical form, it MUST treat this as a protocol error. If the implementation chooses to handle this protocol error in the spirit of the robustness principle (being liberal in what it accepts), it MUST remove all but one of the duplicate RR(s) for the purposes of calculating the canonical form of the RRset."

Requirements:

  1. Implement the "set" interface.
  2. Get / set the RRSet ttl. Default to "unset"
  3. Add new dns.IRecords to the RRSet. Hash based on canonical byte format.
    • Duplicate records will be silently dropped.
    • If RRSET TTL is None, it will be set to that of the record with the lowest TTL.
    • A flag to force an raise an exception if subsequent records have a different TTL.
    • Or perhaps better to return error codes as FlagConstants - so that the caller doesn't *have* to handle the exception if they're not interested. See "Receiving RRSETs below.
    • Mark the RRSet as "inconsistent"
  4. dns.RRSet will implement dns.IEncodable.encode
  5. "encode" will overwrite the TTLs of all records before they are encoded and return the encoded bytes.

dns.RRContainer for managing records as RRSETs in dns.EDNSMessage.answers/authority/additional

Instead of lists, dns.EDNSMessage.answers will be a special container which can be queried for records by name, or by name and type and consolidates records having the same name and type before returning RRSet instances.

Requirements:

  • Access records from one of the EDNSMessage sections (answers, authority, and additional) as RRSet objects.
  • RRSet objects have a flag indicating the consistency of their record TTLs
  • Add single records or lists of records to a specific section of an EDNSMessage.

dns.EDNSMessage Selective truncate during encoding

"The TC bit should be set in responses only when an RRSet is required as a part of the response, but could not be included in its entirety."

dns.Message currently truncates messages based on the combined length of the answer, auth, and additional sections. Instead it (or something higher up) should look at the length of the answers and only set the truncate flag if the entire answers RRSET is greater than the maxPayloadSize. If there is room, the auth and additional records can also be included, but entire RRSETs must be included or none.

Requirements:

  • During encoding, set the trunc bit if wire message containing only answers would exceed the chosen max UDP payload size.
  • Add as many full RRSets to authority and additional sections as possible without exceeding the max UDP payload size.

client.EDNSResolver RRSet handling

"Should a client receive a response containing RRs from an RRSet with differing TTLs, it should treat this as an error. If the RRSet concerned is from a non-authoritative source for this data, the client should simply ignore the RRSet, and if the values were required, seek to acquire them from an authoritative source. Clients that are configured to send all queries to one, or more, particular servers should treat those servers as authoritative for this purpose. Should an authoritative source send such a malformed RRSet, the client should treat the RRs for all purposes as if all TTLs in the RRSet had been set to the value of the lowest TTL in the RRSet. In no case may a server send an RRSet with TTLs not all equal."

Requirements:

  • Drop all records and try a different server if a *recursive* query yields RRSets containing inconsistent TTLs.
  • Accept and return a consistent RRSet if a *authoritative* query yields inconsistent RRSets.

Caching RRSETs

twisted.names.cache should follow the RRSET ranking guidance when serving and replacing items in its cache.

  • https://tools.ietf.org/html/rfc2181#section-5.4
  • "Servers must never merge RRs from a response with RRs in their cache to form an RRSet. If a response contains data that would form an RRSet with data in a server's cache the server must either ignore the RRs in the response, or discard the entire RRSet currently in the cache, as appropriate."
  • https://tools.ietf.org/html/rfc2181#section-5.4.1
  • "When considering whether to accept an RRSet in a reply, or retain an RRSet already in its cache instead, a server should consider the relative likely trustworthiness of the various data"

Requirements:

  • TODO

Serving RRSETs

twisted.names.dns.Message should follow the guidance in RFC2181 regarding the handling of TTLs for records in the same RRSET. Alternatively consider handling this higher up in t.n.authority.FileAuthority, t.n.cache, t.n.resolver etc.

Requirements:

  • TODO

DNSSEC

Some of this work has already been implemented by BobNovas in two large patches attached to original tickets: #5450, #5453, #5454. Look there before implementing anything from scratch.

New DNSSEC Records and Lookup Methods

  1. DNSSEC introduces six new resource record types. Each new record type will require a new dns.Record subclass and a new lookupMethod added to t.i.interfaces.IResolver, t.n.common.ResolverBase and a corresponding free function in t.n.client.
    1. #6664 DNSKey https://tools.ietf.org/html/rfc4034#section-2
    2. #6665 RRSIG https://tools.ietf.org/html/rfc4034#section-3
    3. NSEC https://tools.ietf.org/html/rfc4034#section-4
    4. DS https://tools.ietf.org/html/rfc4034#section-5
    5. NSEC3 https://tools.ietf.org/html/rfc5155#section-3
    6. NSEC3Param https://tools.ietf.org/html/rfc5155#section-4
  2. For ease of review, this work can be split into six tickets.
  3. These new records can be implemented independently of EDNS and independently of DNSSEC validation and new DNSSEC related message headers.
  4. Initially, this will allow twisted.names clients to explicitly request these DNSSEC related records.
  5. lookupZone will return DNSSEC related records when transferring from DNSSEC authoritative servers.
  6. t.n.secondary.SecondaryAuthority will download DNSSEC records and serve them when queries specifically ask for them by type.
  7. t.n.authority.FileAuthority will load DNSSEC records and serve them when queries specifically ask for them by type.

Security-aware Non-validating Client

A twisted.names.client will be able to generate EDNS queries with one or both the DO bit and the AD bit set.

It will examine the state of the AD bit in the response to determine whether the upstream resolver claims to have validated the records in the response.

If the DO bit was set, it will expect to receive DNSSEC related records in the response.

https://tools.ietf.org/html/rfc4035#section-4.9

Validating Client

A twisted.names.client which sends DO + CD flagged queries and performs its own validation of the returned DNSSEC signatures.

TODO: needs more thought.

Validating Recursive / Forwarding Server

This can use the Validating client API above, but may need to do some processing of answers based on the query flags.

TODO: needs more thought.

DNSSEC Aware Authoritative Server

A twisted.names.authority API which knows where and when to include RRSIG, DNSKEY, NSEC, NSEC3 records etc with responses.

The actual generation of the DNSSEC records can be performed using external tools such as dnssec-signzone

TODO: needs more thought.

RFCs

General

  1. 1034:DOMAIN NAMES - CONCEPTS AND FACILITIES
  2. 1035: DOMAIN NAMES - IMPLEMENTATION AND SPECIFICATION
  3. 2181: Clarifications to the DNS Specification
  4. 4697: Observed DNS Resolution Misbehavior
  5. 5625: DNS Proxy Implementation Guidelines

EDNS

  1. 6891: Extension Mechanisms for DNS (EDNS(0))

DNSSEC

  1. 4033: DNS Security Introduction and Requirements
  2. 4034: Resource Records for the DNS Security Extensions
  3. 4035: Protocol Modifications for the DNS Security Extensions
  4. 5155: DNS Security (DNSSEC) Hashed Authenticated Denial of Existence
  5. 6781: DNSSEC Operational Practices, Version 2
  6. 6840: Clarifications and Implementation Notes for DNS Security (DNSSEC)
  7. 3833: Threat Analysis of the Domain Name System (DNS)

Existing Implementations

Re-use of existing Python DNS libraries (or C libraries with Python bindings) will be considered according to the following criteria:

  1. MIT compatible licence
  2. Actively maintained
  3. Full test coverage

However the potential time savings from reusing an existing library will be weighed against the overhead of introducing an additional Twisted install dependency.

Examples of existing libraries / implementations:

Other sources:

Dissemination Efforts

  1. Conference talk "Twisted Names" at EuroPython June/July 2014: http://europython.eu
  1. Fortnightly progress reports via blog syndicated to http://planet.twistedmatrix.com/ and http://planet.python.org/
  1. Submit "Introduction to Twisted Names" article on CircleID: http://www.circleid.com/ or another DNS industry news site.
  1. Attend development sprints to continue the development, and to introduce new contributors to the capabilities of Twisted Names DNSSEC support.

People and roles

  • Richard Wall: will be coordinating the project
  • Tom Prince will provide code reviews.
  • The project will be building on a series of patches originally contributed by Bob Novas and Phil Mayers. Bob and Phil have been invited to help with design reviews, code reviews and development.

Risks

  1. Review delays - Finding knowledgeable reviewers willing to do prompt paid reviews.
  1. Unexpected groundwork - Twisted Names has not been particularly well maintained and there are a number of known bugs. I have included some of these in the milestones, but there may be other unforeseen problems that need to be fixed before continuing with the EDNS / DNSSEC work.
  1. Blocked by dependencies - Twisted currently depends on PyOpenSSL and PyCrypto. I need to find out whether these expose all the necessary APIs required for DNSSEC implementation. PyOpenSSL is currently maintained by one of the core Twisted developers so it shouldn't be a problem to add any missing parts.
  1. Changes in personal circumstances - new jobs, etc.
  1. Backwards compatibility - difficulties combining the new EDNS and DNSSEC APIs with existing APIs without breaking backwards compatibility.
  1. Lack of uptake - DNSSEC is still not widely adopted and not widely understood, so it may take some time to convince maintainers to depend on it.

Mission and Aim

To implement a client DNSSEC verification API in Twisted.

This project will lay foundations that will eventually allow end-to-end DNSSEC verification in all the core Twisted networking components, including Twisted Conch (SSH), Mail (SMTP, POP3), Perspective Broker (RPC), Web (HTTP, XML-RPC, SOAP), Words (XMPP, IRC).

This foundation work will encourage the development of end-to-end DNSSEC verification in many of the Open Source and commercial projects built on top of Twisted.

For example, Twisted is used in the following important open source projects:

  1. Apple Calendar Server: a standards-compliant calendar and contacts server implementing the CalDAV and CardDAV protocols. http://trac.calendarserver.org/browser/CalendarServer/trunk/HACKING
  1. Buildbot: a continuous integration framework for automating software build, test, and release processes. http://docs.buildbot.net/latest/manual/installation.html#requirements
  1. Convergence: a secure, distributed replacement for the Certificate Authority System. https://github.com/moxie0/Convergence/blob/master/server/INSTALL
  1. Tahoe-LAFS: a decentralized cloud storage system with provider-independent security. https://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-deps/
  1. Tor OONI: Open Observatory of Network Interference - a tool for performing internet censorship measurements. https://gitweb.torproject.org/ooni-probe.git/blob_plain/HEAD:/requirements.txt
  1. Zenoss: a network management platform for monitoring availability, inventory/configuration, performance, and events. http://dev.zenoss.org/trac/browser/trunk/inst/externallibs

There are hundreds of other open source projects which either extend Twisted or which allow Twisted to be integrated with third party software and services.

Twisted is a core component in various Unix operating systems eg Debian, Ubuntu, Fedora and is packaged for others too.

Twisted is used in the services and software of various high profile companies such as Apple, Google and Rackspace.

The most compelling reason for adding DNSSEC support to Twisted, is that it will make end-to-end DNSSEC verification available to hundreds of other projects that are built on top of Twisted.

By integrating the proposed DNSSEC verification API with Twisted's existing client endpoint APIs it should be straight forward for application developers to make DNSSEC verification available in their software.

It is also worth noting that the proposed DNSSEC verification mechanism will be fully supported on all Twisted's target platforms, including Microsoft Windows. Which means that Windows based Twisted applications will also be able to benefit from end-to-end DNSSEC verification.

Finally it is worth noting that Twisted Names is written entirely in Python. Its simple / readable code makes Twisted Names ideal for DNS prototyping and experimentation. And with the addition of EDNS(0) and DNSSEC Twisted Names will be an even more flexible platform with which to implement robust prototypes of new DNS standards.

Continuations

We anticipate a number of interesting future projects which will be made possible by the foundation work described in this proposal:

  1. Integration of DNSSEC into Twisted's new "Happy Eyeballs" hostname client endpoint, allowing secure transparent client connections from a dual-stack IPv4 / IPv6 host.
  1. Add TLSA / DANE support to Twisted Web Agent, allowing Twisted HTTP clients to additionally validate certificates using signed DNS records.
  1. Add SSHFP support to Twisted Conch, allowing Twisted SSH clients to validate the public keys of a remote SSH server using signed DNS records.
  1. Add S/MIME and / or SMTP/TLSA support to Twisted Mail, allowing Twisted SMTP clients to validate remote server certificates using signed DNS records.
  1. A Twisted DNS stub resolver which communicates securely with a Twisted DNSSEC validating server via an SSL/TLS or SSH transport. For situations where a Twisted client can't do DNSSEC verification its self.
  1. A Twisted DNSSEC management server with an integrated SSH / HTTPS REST API for receiving and signing DS keys from the servers for delegated child zones.