- EDNS Plan
- #6680 Add Authentic Data and Checking Disabled flags
- #5668 dns.OPTHeader
- #5675 dns.EDNSMessage
- #6839 dns.EDNSStreamProtocol and dns.EDNSDatagramProtcol
- #6840 client.Resolver protocol override options
- #5670 client.EDNSResolver Fixed UDP Payload Size
- client.EDNSResolver with Automatic Payload Size Selection and Detection
- #5669 server.EDNSServerFactory
- twistd dns plugin - EDNS(0) by default
- Variable EDNS0 Option Codes (OPT)
- dns.Name.canonical - Canonical form and order of DNS names
- dns.IRecord.canonical - Canonical Form and Order of Resource Records
- dns.RRContainer for managing records as RRSETs in …
- dns.EDNSMessage Selective truncate during encoding
- client.EDNSResolver RRSet handling
- Caching RRSETs
- Serving RRSETs
- DNSSEC Record Types and Lookup Methods
- DNSSEC Client
Twisted includes a comprehensive set of DNS components, collectively known as twisted Names.
This is a plan for the development of EDNS(0) and DNSSEC support in Twisted Names.
- Contributions will be submitted as MIT licensed patches to the Twisted project.
- All new code will be accompanied by full API documentation and new public APIs must also have accompanying narrative documentation.
- All code must have unit tests and there must be 100% statement and branch coverage as measured by the python-coverage tool.
- New public APIs will be as narrow as possible, to minimise future backwards compatibility complications.
- Zero changes to existing public APIs. (if possible)
- Existing classes will be wrapped where appropriate and in other cases, shared functionality will be extracted and re-factored for reuse in new classes.
- Composition, not inheritance
- Old sub-optimal APIs will be deprecated and eventually removed after the introduction new EDNS and DNSSEC APIs.
- NLnet Foundation: This project was made possible with funding from the DNSSEC fund operated by NLnet foundation (http://nlnet.nl/dnssec). If you think DNSSEC is important, please make a donation to this fund.
- Bob Novas and Phil Mayers: Who contributed the original EDNS(0) and DNSSEC patches on which this work is based.
- A twistd dns server capable of responding to EDNS(0) clients and issuing EDNS(0) requests to upstream DNS servers.
- A client API for issuing EDNS(0) requests with a fixed max UDP payload size.
- A message parsing API capable of encoding and decoding DNS messages containing EDNS(0) specific fields and pseudo records.
Add two new fields to the existing dns.Message class
- Set / get AD and CD field
- Byte encode / decode AD and CD field
Parse and construct OPT Pseudo-RR
- Get / set fixed OPT record fields.
- Byte encode / decode EDNS OPT records.
- Convert to / from dns.RRHeader (the existing non-EDNS record header class)
- Get / set variable OPT record fields.
- Byte encode / decode variable OPT record fields.
- Convert to / from dns.UnknownRecord (the existing non-EDNS record payload class for the unknown record payload types.)
Parse and construct EDNS messages
- Get / set all non-EDNS message fields
- Get / set all new EDNS(0) message fields
- Byte encoding / decoding
- During encoding, EDNS specific fields will be used to generate a dns.OPTHeader instance which will be added to the end of the additional records list.
- During decoding, an OPT pseudo record found in the additional section will be extracted, parsed and its fields will be used to populate the EDNS specific attributes of the EDNSMessage instance.
- Get / set extended RCODE
- A mechanism for getting protocol errors encountered during parsing. eg
- "If a query message with more than one OPT RR is received, a FORMERR (RCODE=1) MUST be returned"
- Convert to / from dns.Message (the existing non-EDNS message class)
- We will not implement an API for manipulating OPT variable fields at this stage. See "Variable EDNS0 Option Codes" below.
The existing dns.DNSDatagramProtocol and DNSProtocol (TCP) are hardcoded to use dns.Message for decoding and encoding wire messages.
The simplest solution would be to add a new "messageFactory" constructor argument which allows us to supply a curried dns.EDNSMessage instance whose EDNS specific constructor arguments have been preassigned.
Alternatively it might be better to introduce new narrower EDNS Protocol APIs which wrap DNSProtocol, and DNSDatagramProtocol and hide many of their ugly implementation details.
The wrappers can override the "writeMessage" method of the original protocols as demonstrated here:
- DNS datagram and stream protocol implementations which can use dns.EDNSMessage for decoding and encoding wire messages.
- Get / set EDNS specific options which will be passed to the construct the EDNSMessage instance responsible for encoding and decoding.
client.Resolver is currently hard coded to use dns.DNSDatagramProtocol and dns.DNSProtocol (TCP) via client.DNSClientFactory.
Allow the caller to supply alternative protocol factories.
- client.Resolver constructor will accept a "datagramProtocolFactory" and "streamProtocolFactory" arguments, which will default to existing factories but which will allow dns.EDNSDatagramProtocol to be supplied instead.
Start with a basic EDNSResolver with a fixed UDP payload size and which only does TCP fallback.
- An implementation of t.i.interfaces.IResolver which sends EDNS(0) queries by
- Wraps client.Resolver
- Supplies an EDNSMessage factory to allow setting EDNS specific fields for all queries.
- Get / set AD, CD
- Get / set DO field. Default to "unset"
- Get / set VERSION. Default to 0.
- Get / set maximum UDP payload size.
- Default to 4096
- Detect servers which do not support EDNS(0) and fall back.
- ONLY fallback if DO is "unset"
- non-EDNS UDP query
- TCP query
- We will not implement an API for manipulating OPT variable fields at this stage. See "Variable EDNS0 Option Codes" below.
client.EDNSResolver with Automatic Payload Size Selection and Detection
Extend client.EDNSResolver to detect timeouts and fragmentation caused by UDP payload size limits of the server and intermediate devices. Automatically re-issue with successively smaller advertised payload sizes, possibly starting with the server advertised payload size found in the first reply.
Consider postponing this. It's probably not required for basic DNSSEC client support.
"A requestor MAY choose to implement a fallback to smaller advertised sizes to work around firewall or other network limitations."
- Issue parallel queries with a small delay between each and with successively smaller advertised UDP payload sizes.
- Cache and re-use the detected maximum payload size for each server.
- Fallback to TCP query.
- "Due to transaction overhead, it is not recommended to advertise an architectural limit as a maximum UDP payload size"
- "Values of less than 512 bytes MUST be treated as equal to 512 bytes."
server.DNSServerFactory is responsible for coordinating datagram and stream listening ports and their protocols.
The protocols dispatch their received queries to DNSServerFactory and it in turn dispatches the queries to one or more authoritative, caching or recursive IResolver instances.
It then constructs a response message, populates the answers, authority and additional lists and supplies it to the write method of the appropriate protocol instance.
DNSServerFactory is also responsible for enforcing policy, such as checking the origin of queries or notify messages.
One problem with the existing implementation is that it uses the incoming message and modifies it in place before sending it back to the client. This results in various request fields being returned to the client as if they are the server's chosen response fields.
This is completely wrong for the EDNS payload size and the new AD and CD fields.
A related problem is that it also includes the client's original OPT record in error responses #6645
So here we will implement an EDNSServerFactory somehow overrides the "messageReceived" method in order to enforce correct server max UDP payload size.
Instead of re-using the incoming message instance, we will instead construct a new instance and carefully choose the field values based on the servers configuration.
- Get / set maximum UDP payload size.
- Send EDNS0 responses to EDNS0 queries
- Do not send EDNS0 responses to standard DNS queries.
- Respond clients using their advertised UDP payload size. (up to the server maximum).
- Limit additional RRSETs so that responses fit the client max UDP payload size or mark response Messages truncated.
- Set extended EDNS RCODEs in responses where appropriate.
twistd dns plugin - EDNS(0) by default
At this point we should have enough infrastructure to allow the twistd dns plugin to correctly respond to EDNS(0) queries.
If configured as a forwarding resolver, it will be able to issue EDNS(0) queries to the upstream servers.
For compatibility we may choose to make it respond to clients with non-DNS messages, but using the new EDNSMessage API.
Then at a later date we can add an --edns flag to allow the EDNS features to be turned on or off. Or we could do it all in this ticket.
This isn't strictly needed for the goal of having a DNSSEC validating client, but it will be a nice way in which Twisted users can quickly benefit from the new EDNS APIs.
Variable EDNS0 Option Codes (OPT)
An OPT record payload can contain one or more variable fields. The list of defined fields is here:
One example is the EDNS(0) owner option which allows a DNS proxy (recursive or forwarding resolver) to include information about the origin of the client query (the client subnet), when it queries an authoritative server. An authoritative server can use this information to do DNS based global load balancing.
Each of these variable fields would need it's own API so we need to give thought to how the EDNSResolver can be made extensible enough to support these.
- A client API capable of correctly handling RRSETs; according to the rules defined in RFC2181.
- A server API capable of generating DNS response messages containing the correct RRSETs, in the correct sections of the message. And truncating messages correctly based on the rules in RF2181.
- A DNS message parsing API capable of handling RRSETs in each of its sections.
- An API for encoding and decoding RRSETs; groups of records having the same name and type.
- The ability to sort DNS records and names based on the canonical form described in rfc4034.
DNSSEC relies on stable RRSETs. eg for signatures, for predictable caching of records and signatures and for predictable truncation of large DNSSEC responses.
RRSETs must be arranged in canonical order before their signatures are calculated / verified.
These tickets will introduce algorithms for sorting records according to the rules described in https://tools.ietf.org/html/rfc4034#section-6.
Sorting appears to be the responsibility of the verifying client not the server.
The fact that Bind has config options for changing the order of RRSETS on the server side and the client side suggests that the canonical ordering of records should be done only for the purpose of DNSSEC validation. It should probably not change the order of records returned by various IResolver methods.
dns.Name.canonical - Canonical form and order of DNS names
- A "canonical" property will be added to dns.Name which will return the name in RFC4034 canonical form as bytes. + https://tools.ietf.org/html/rfc4034#section-6.1
- Names can be sorted canonically by passing the "canonical" attribute name as the "key" to sorted
dns.IRecord.canonical - Canonical Form and Order of Resource Records
- All IRecord implementations will be given a "canonical" method which returns bytes representing the RR in canonical wire format.
- A "canonical" method will be added to dns.IRecord
- Records can be sorted canonically by passing the "canonical" attribute name as the "key" to sorted
"An RRset is not allowed to contain duplicate records (multiple RRs with the same owner name, class, type, and RDATA). Therefore, if an implementation detects duplicate RRs when putting the RRset in canonical form, it MUST treat this as a protocol error. If the implementation chooses to handle this protocol error in the spirit of the robustness principle (being liberal in what it accepts), it MUST remove all but one of the duplicate RR(s) for the purposes of calculating the canonical form of the RRset."
- Implement the "set" interface.
- Get / set the RRSet ttl. Default to "unset"
- Add new dns.IRecords to the RRSet. Hash based on canonical byte format.
- Duplicate records will be silently dropped.
- If RRSET TTL is None, it will be set to that of the record with the lowest TTL.
- A flag to force an raise an exception if subsequent records have a different TTL.
- Or perhaps better to return error codes as FlagConstants - so that the caller doesn't *have* to handle the exception if they're not interested. See "Receiving RRSETs below.
- Mark the RRSet as "inconsistent"
- dns.RRSet will implement dns.IEncodable.encode
- "encode" will overwrite the TTLs of all records before they are encoded and return the encoded bytes.
dns.RRContainer for managing records as RRSETs in dns.EDNSMessage.answers/authority/additional
Instead of lists, dns.EDNSMessage.answers will be a special container which can be queried for records by name, or by name and type and consolidates records having the same name and type before returning RRSet instances.
- Access records from one of the EDNSMessage sections (answers, authority, and additional) as RRSet objects.
- RRSet objects have a flag indicating the consistency of their record TTLs
- Add single records or lists of records to a specific section of an EDNSMessage.
dns.EDNSMessage Selective truncate during encoding
"The TC bit should be set in responses only when an RRSet is required as a part of the response, but could not be included in its entirety."
dns.Message currently truncates messages based on the combined length of the answer, auth, and additional sections. Instead it (or something higher up) should look at the length of the answers and only set the truncate flag if the entire answers RRSET is greater than the maxPayloadSize. If there is room, the auth and additional records can also be included, but entire RRSETs must be included or none.
- During encoding, set the trunc bit if wire message containing only answers would exceed the chosen max UDP payload size.
- Add as many full RRSets to authority and additional sections as possible without exceeding the max UDP payload size.
client.EDNSResolver RRSet handling
"Should a client receive a response containing RRs from an RRSet with differing TTLs, it should treat this as an error. If the RRSet concerned is from a non-authoritative source for this data, the client should simply ignore the RRSet, and if the values were required, seek to acquire them from an authoritative source. Clients that are configured to send all queries to one, or more, particular servers should treat those servers as authoritative for this purpose. Should an authoritative source send such a malformed RRSet, the client should treat the RRs for all purposes as if all TTLs in the RRSet had been set to the value of the lowest TTL in the RRSet. In no case may a server send an RRSet with TTLs not all equal."
- Drop all records and try a different server if a *recursive* query yields RRSets containing inconsistent TTLs.
- Accept and return a consistent RRSet if a *authoritative* query yields inconsistent RRSets.
twisted.names.cache should follow the RRSET ranking guidance when serving and replacing items in its cache.
- "Servers must never merge RRs from a response with RRs in their cache to form an RRSet. If a response contains data that would form an RRSet with data in a server's cache the server must either ignore the RRs in the response, or discard the entire RRSet currently in the cache, as appropriate."
- "When considering whether to accept an RRSet in a reply, or retain an RRSet already in its cache instead, a server should consider the relative likely trustworthiness of the various data"
twisted.names.dns.Message should follow the guidance in RFC2181 regarding the handling of TTLs for records in the same RRSET. Alternatively consider handling this higher up in t.n.authority.FileAuthority, t.n.cache, t.n.resolver etc.
- "the use of differing TTLs in an RRSet is hereby deprecated, the TTLs of all RRs in an RRSet must be the same"
- "In no case may a server send an RRSet with TTLs not all equal."
- "A Resource Record Set should only be included once in any DNS reply. It may occur in any of the Answer, Authority, or Additional Information sections, as required. However it should not be repeated in the same, or any other, section, except where explicitly required by a specification."
DNSSEC Record Types and Lookup Methods
- A Twisted based version of dig; a command line tool capable of issuing DNSSEC queries. This will demonstrate the use of the client APIs below.
- A twistd authoritative DNS server capable of loading and serving DNSSEC records.
- A twistd recursive / forwarding proxy server capable of issuing DNSSEC records on behalf of a client.
- Client API capable of issuing requests for DNSSEC records
- Client API capable of decoding DNSSEC records.
- Record APIs for encoding and decoding DS, DNSKEY, RRSIG, NSEC, NSEC3, NSECPARAM records.
DNSSEC introduces six new resource record types. Each new record type will require a new dns.Record subclass and a new lookupMethod added to t.i.interfaces.IResolver, t.n.common.ResolverBase and a corresponding free function in t.n.client.
- For ease of review, this work can be split into six tickets.
- These new records can be implemented independently of EDNS and independently of DNSSEC validation and new DNSSEC related message headers.
- Initially, this will allow twisted.names clients to explicitly request these DNSSEC related records.
- lookupZone will return DNSSEC related records when transferring from DNSSEC authoritative servers.
- t.n.secondary.SecondaryAuthority will download DNSSEC records and serve them when queries specifically ask for them by type.
- t.n.authority.FileAuthority will load DNSSEC records and serve them when queries specifically ask for them by type.
- A HostnameEndpoint wrapper capable of using a validating or non-validating stub resolver and which returns DNS lookup errors in the event of failed DNS validation.
- twistd dns will be capable of operating as a validating security-aware recursive resolver which can be used by other non-validating stub resolvers.
- A non-validating security-aware stub resolver
- A validating security-aware stub resolver
Security-aware Non-validating Client
A twisted.names.client will be able to generate EDNS queries with one or both the DO bit and the AD bit set.
It will examine the state of the AD bit in the response to determine whether the upstream resolver claims to have validated the records in the response.
If the DO bit was set, it will expect to receive DNSSEC related records in the response.
A twisted.names.client which sends DO + CD flagged queries and performs its own validation of the returned DNSSEC signatures.
TODO: needs more thought.
Validating Recursive / Forwarding Server
This can use the Validating client API above, but may need to do some processing of answers based on the query flags.
TODO: needs more thought.
DNSSEC Aware Authoritative Server
A twisted.names.authority API which knows where and when to include RRSIG, DNSKEY, NSEC, NSEC3 records etc with responses.
The actual generation of the DNSSEC records can be performed using external tools such as dnssec-signzone
TODO: needs more thought.
- A twistd dns authoritative server capable of loading and serving TLSA records.
- A Twisted web client Agent wrapper which performs TLSA lookup and verification of a server certificate.
- A HostnameClientEndpoint which performs TLSA lookup and verification of a server certificate.
- A command line tool for debugging TLSA records and for verifying a certificate file against a domain name.
- A TLSA Record class for encoding and decoding TLSA bytes.
- A TLSA lookup method which accepts port, protocol and hostname and constructs a suitable TLSA domain name.
TLSA lookup method
TLSA verification tool
TLSA Hostname Endpoint
TLSA Web Agent
- 1034:DOMAIN NAMES - CONCEPTS AND FACILITIES
- 1035: DOMAIN NAMES - IMPLEMENTATION AND SPECIFICATION
- 2181: Clarifications to the DNS Specification
- 4697: Observed DNS Resolution Misbehavior
- 5625: DNS Proxy Implementation Guidelines
- 4033: DNS Security Introduction and Requirements
- 4034: Resource Records for the DNS Security Extensions
- 4035: Protocol Modifications for the DNS Security Extensions
- 5155: DNS Security (DNSSEC) Hashed Authenticated Denial of Existence
- 6781: DNSSEC Operational Practices, Version 2
- 6840: Clarifications and Implementation Notes for DNS Security (DNSSEC)
- 3833: Threat Analysis of the Domain Name System (DNS)
- 6394: Use Cases and Requirements for DNS-Based Authentication of Named Entities (DANE)
- 6698: DANE TLSA - The DNS-Based Authentication of Named Entities (DANE) for Transport Layer Security (TLS) Protocol
Re-use of existing Python DNS libraries (or C libraries with Python bindings) will be considered according to the following criteria:
- MIT compatible licence
- Actively maintained
- Full test coverage
However the potential time savings from reusing an existing library will be weighed against the overhead of introducing an additional Twisted install dependency.
Examples of existing libraries / implementations:
- Conference talk "Twisted Names" at EuroPython June/July 2014: http://europython.eu
- Fortnightly progress reports via blog syndicated to http://planet.twistedmatrix.com/ and http://planet.python.org/
- Submit "Introduction to Twisted Names" article on CircleID: http://www.circleid.com/ or another DNS industry news site.
- Attend development sprints to continue the development, and to introduce new contributors to the capabilities of Twisted Names DNSSEC support.
People and Roles
- Richard Wall: will be coordinating the project
- Tom Prince will provide code reviews.
- The project will be building on a series of patches originally contributed by Bob Novas and Phil Mayers. Bob and Phil have been invited to help with design reviews, code reviews and development.
- Review delays - Finding knowledgeable reviewers willing to do prompt paid reviews.
- Unexpected groundwork - Twisted Names has not been particularly well maintained and there are a number of known bugs. I have included some of these in the milestones, but there may be other unforeseen problems that need to be fixed before continuing with the EDNS / DNSSEC work.
- Blocked by dependencies - Twisted currently depends on PyOpenSSL and PyCrypto. I need to find out whether these expose all the necessary APIs required for DNSSEC implementation. PyOpenSSL is currently maintained by one of the core Twisted developers so it shouldn't be a problem to add any missing parts.
- Changes in personal circumstances - new jobs, etc.
- Backwards compatibility - difficulties combining the new EDNS and DNSSEC APIs with existing APIs without breaking backwards compatibility.
- Lack of uptake - DNSSEC is still not widely adopted and not widely understood, so it may take some time to convince maintainers to depend on it.
Mission and Aim
To implement a client DNSSEC verification API in Twisted.
This project will lay foundations that will eventually allow end-to-end DNSSEC verification in all the core Twisted networking components, including Twisted Conch (SSH), Mail (SMTP, POP3), Perspective Broker (RPC), Web (HTTP, XML-RPC, SOAP), Words (XMPP, IRC).
This foundation work will encourage the development of end-to-end DNSSEC verification in many of the Open Source and commercial projects built on top of Twisted.
For example, Twisted is used in the following important open source projects:
- Apple Calendar Server: a standards-compliant calendar and contacts server implementing the CalDAV and CardDAV protocols. http://trac.calendarserver.org/browser/CalendarServer/trunk/HACKING
- Buildbot: a continuous integration framework for automating software build, test, and release processes. http://docs.buildbot.net/latest/manual/installation.html#requirements
- Convergence: a secure, distributed replacement for the Certificate Authority System. https://github.com/moxie0/Convergence/blob/master/server/INSTALL
- Tahoe-LAFS: a decentralized cloud storage system with provider-independent security. https://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-deps/
- Tor OONI: Open Observatory of Network Interference - a tool for performing internet censorship measurements. https://gitweb.torproject.org/ooni-probe.git/blob_plain/HEAD:/requirements.txt
- Zenoss: a network management platform for monitoring availability, inventory/configuration, performance, and events. http://dev.zenoss.org/trac/browser/trunk/inst/externallibs
There are hundreds of other open source projects which either extend Twisted or which allow Twisted to be integrated with third party software and services.
Twisted is a core component in various Unix operating systems eg Debian, Ubuntu, Fedora and is packaged for others too.
Twisted is used in the services and software of various high profile companies such as Apple, Google and Rackspace.
The most compelling reason for adding DNSSEC support to Twisted, is that it will make end-to-end DNSSEC verification available to hundreds of other projects that are built on top of Twisted.
By integrating the proposed DNSSEC verification API with Twisted's existing client endpoint APIs it should be straight forward for application developers to make DNSSEC verification available in their software.
It is also worth noting that the proposed DNSSEC verification mechanism will be fully supported on all Twisted's target platforms, including Microsoft Windows. Which means that Windows based Twisted applications will also be able to benefit from end-to-end DNSSEC verification.
Finally it is worth noting that Twisted Names is written entirely in Python. Its simple / readable code makes Twisted Names ideal for DNS prototyping and experimentation. And with the addition of EDNS(0) and DNSSEC Twisted Names will be an even more flexible platform with which to implement robust prototypes of new DNS standards.
We anticipate a number of interesting future projects which will be made possible by the foundation work described in this proposal:
- Integration of DNSSEC into Twisted's new "Happy Eyeballs" hostname client endpoint, allowing secure transparent client connections from a dual-stack IPv4 / IPv6 host.
- Add SSHFP support to Twisted Conch, allowing Twisted SSH clients to validate the public keys of a remote SSH server using signed DNS records.
- Add S/MIME and / or SMTP/TLSA support to Twisted Mail, allowing Twisted SMTP clients to validate remote server certificates using signed DNS records.
- A Twisted DNS stub resolver which communicates securely with a Twisted DNSSEC validating server via an SSL/TLS or SSH transport. For situations where a Twisted client can't do DNSSEC verification its self.
- A Twisted DNSSEC management server with an integrated SSH / HTTPS REST API for receiving and signing DS keys from the servers for delegated child zones.