Tag Archives: DNS

DNS over TLS in FreeBSD 12

With the arrival of OpenSSL 1.1.1, an upgraded Unbound, and some changes to the setup and init scripts, FreeBSD 12.0, currently in beta, now supports DNS over TLS out of the box.

DNS over TLS is just what it sounds like: DNS over TCP, but wrapped in a TLS session. It encrypts your requests and the server’s replies, and optionally allows you to verify the identity of the server. The advantages are protection against eavesdropping and manipulation of your DNS traffic; the drawbacks are a slight performance degradation and potential firewall traversal issues, as it runs over a non-standard port (TCP port 853) which may be blocked on some networks. Let’s take a look at how to set it up.

Basic setup

As a simple test case, let’s set up our 12.0-ALPHA10 VM to use Cloudflare’s DNS service:

# uname -r
12.0-ALPHA10
# cat >/etc/rc.conf.d/local_unbound <<EOF
local_unbound_enable="YES"
local_unbound_tls="YES"
local_unbound_forwarders="1.1.1.1@853 1.0.0.1@853"
EOF
# service local_unbound start
Performing initial setup.
destination:
/var/unbound/forward.conf created
/var/unbound/lan-zones.conf created
/var/unbound/control.conf created
/var/unbound/unbound.conf created
/etc/resolvconf.conf not modified
Original /etc/resolv.conf saved as /var/backups/resolv.conf.20181021.192629
Starting local_unbound.
Waiting for nameserver to start... good
# host www.freebsd.org
www.freebsd.org is an alias for wfe0.nyi.freebsd.org.
wfe0.nyi.freebsd.org has address 96.47.72.84
wfe0.nyi.freebsd.org has IPv6 address 2610:1c1:1:606c::50:15
wfe0.nyi.freebsd.org mail is handled by 0 .

Note that this is not a configuration you want to run in production—we will come back to this later.

Performance

The downside of DNS over TLS is the performance hit of the TCP and TLS session setup and teardown. We demonstrate this by flushing our cache and (rather crudely) measuring a cache miss and a cache hit:

# local-unbound-control reload
ok
# time host www.freebsd.org >x
host www.freebsd.org > x 0.00s user 0.00s system 0% cpu 0.553 total
# time host www.freebsd.org >x
host www.freebsd.org > x 0.00s user 0.00s system 0% cpu 0.005 total

Compare this to querying our router, a puny Soekris net5501 running Unbound 1.8.1 on FreeBSD 11.1-RELEASE:

# time host www.freebsd.org gw >x
host www.freebsd.org gw > x 0.00s user 0.00s system 0% cpu 0.232 total
# time host www.freebsd.org 192.168.144.1 >x
host www.freebsd.org gw > x 0.00s user 0.00s system 0% cpu 0.008 total

or to querying Cloudflare directly over UDP:

# time host www.freebsd.org 1.1.1.1 >x      
host www.freebsd.org 1.1.1.1 > x 0.00s user 0.00s system 0% cpu 0.272 total
# time host www.freebsd.org 1.1.1.1 >x
host www.freebsd.org 1.1.1.1 > x 0.00s user 0.00s system 0% cpu 0.013 total

(Cloudflare uses anycast routing, so it is not so unreasonable to see a cache miss during off-peak hours.)

This clearly shows the advantage of running a local caching resolver—it absorbs the cost of DNSSEC and TLS. And speaking of DNSSEC, we can separate that cost from that of TLS by reconfiguring our server without the latter:

# cat >/etc/rc.conf.d/local_unbound <<EOF
local_unbound_enable="YES"
local_unbound_tls="NO"
local_unbound_forwarders="1.1.1.1 1.0.0.1"
EOF
# service local_unbound setup
Performing initial setup.
destination:
Original /var/unbound/forward.conf saved as /var/backups/forward.conf.20181021.205328
/var/unbound/lan-zones.conf not modified
/var/unbound/control.conf not modified
Original /var/unbound/unbound.conf saved as /var/backups/unbound.conf.20181021.205328
/etc/resolvconf.conf not modified
/etc/resolv.conf not modified
# service local_unbound start
Starting local_unbound.
Waiting for nameserver to start... good
# time host www.freebsd.org >x
host www.freebsd.org > x 0.00s user 0.00s system 0% cpu 0.080 total
# time host www.freebsd.org >x
host www.freebsd.org > x 0.00s user 0.00s system 0% cpu 0.004 total

So does TLS add nearly half a second to every cache miss? Not quite, fortunately—in our previous tests, our first query was not only a cache miss but also the first query after a restart or a cache flush, resulting in a complete load and validation of the entire path from the name we queried to the root. The difference between a first and second cache miss is quite noticeable:

# time host www.freebsd.org >x 
host www.freebsd.org > x 0.00s user 0.00s system 0% cpu 0.546 total
# time host www.freebsd.org >x
host www.freebsd.org > x 0.00s user 0.00s system 0% cpu 0.004 total
# time host repo.freebsd.org >x
host repo.freebsd.org > x 0.00s user 0.00s system 0% cpu 0.168 total
# time host repo.freebsd.org >x
host repo.freebsd.org > x 0.00s user 0.00s system 0% cpu 0.004 total

Revisiting our configuration

Remember when I said that you shouldn’t run the sample configuration in production, and that I’d get back to it later? This is later.

The problem with our first configuration is that while it encrypts our DNS traffic, it does not verify the identity of the server. Our ISP could be routing all traffic to 1.1.1.1 to its own servers, logging it, and selling the information to the highest bidder. We need to tell Unbound to validate the server certificate, but there’s a catch: Unbound only knows the IP addresses of its forwarders, not their names. We have to provide it with names that will match the x509 certificates used by the servers we want to use. Let’s double-check the certificate:

# :| openssl s_client -connect 1.1.1.1:853 |& openssl x509 -noout -text |& grep DNS
DNS:*.cloudflare-dns.com, IP Address:1.1.1.1, IP Address:1.0.0.1, DNS:cloudflare-dns.com, IP Address:2606:4700:4700:0:0:0:0:1111, IP Address:2606:4700:4700:0:0:0:0:1001

This matches Cloudflare’s documentation, so let’s update our configuration:

# cat >/etc/rc.conf.d/local_unbound <<EOF
local_unbound_enable="YES"
local_unbound_tls="YES"
local_unbound_forwarders="1.1.1.1@853#cloudflare-dns.com 1.0.0.1@853#cloudflare-dns.com"
EOF
# service local_unbound setup
Performing initial setup.
destination:
Original /var/unbound/forward.conf saved as /var/backups/forward.conf.20181021.212519
/var/unbound/lan-zones.conf not modified
/var/unbound/control.conf not modified
/var/unbound/unbound.conf not modified
/etc/resolvconf.conf not modified
/etc/resolv.conf not modified
# service local_unbound restart
Stopping local_unbound.
Starting local_unbound.
Waiting for nameserver to start... good
# host www.freebsd.org
www.freebsd.org is an alias for wfe0.nyi.freebsd.org.
wfe0.nyi.freebsd.org has address 96.47.72.84
wfe0.nyi.freebsd.org has IPv6 address 2610:1c1:1:606c::50:15
wfe0.nyi.freebsd.org mail is handled by 0 .

How can we confirm that Unbound actually validates the certificate? Well, we can run Unbound in debug mode (/usr/sbin/unbound -dd -vvv) and read the debugging output… or we can confirm that it fails when given a name that does not match the certificate:

# perl -p -i -e 's/cloudflare/cloudfire/g' /etc/rc.conf.d/local_unbound
# service local_unbound setup
Performing initial setup.
destination:
Original /var/unbound/forward.conf saved as /var/backups/forward.conf.20181021.215808
/var/unbound/lan-zones.conf not modified
/var/unbound/control.conf not modified
/var/unbound/unbound.conf not modified
/etc/resolvconf.conf not modified
/etc/resolv.conf not modified
# service local_unbound restart
Stopping local_unbound.
Waiting for PIDS: 33977.
Starting local_unbound.
Waiting for nameserver to start... good
# host www.freebsd.org
Host www.freebsd.org not found: 2(SERVFAIL)

But is this really a failure to validate the certificate? Actually, no. When provided with a server name, Unbound will pass it to the server during the TLS handshake, and the server will reject the handshake if that name does not match any of its certificates. To truly verify that Unbound validates the server certificate, we have to confirm that it fails when it cannot do so. For instance, we can remove the root certificate used to sign the DNS server’s certificate from the test system’s trust store. Note that we cannot simply remove the trust store entirely, as Unbound will refuse to start if the trust store is missing or empty.

While we’re talking about trust stores, I should point out that you currently must have ca_root_nss installed for DNS over TLS to work. However, 12.0-RELEASE will ship with a pre-installed copy.

Conclusion

We’ve seen how to set up Unbound—specifically, the local_unbound service in FreeBSD 12.0—to use DNS over TLS instead of plain UDP or TCP, using Cloudflare’s public DNS service as an example. We’ve looked at the performance impact, and at how to ensure (and verify) that Unbound validates the server certificate to prevent man-in-the-middle attacks.

The question that remains is whether it is all worth it. There is undeniably a performance hit, though this may improve with TLS 1.3. More importantly, there are currently very few DNS-over-TLS providers—only one, really, since Quad9 filter their responses—and you have to weigh the advantage of encrypting your DNS traffic against the disadvantage of sending it all to a single organization. I can’t answer that question for you, but I can tell you that the parameters are evolving quickly, and if your answer is negative today, it may not remain so for long. More providers will appear. Performance will improve with TLS 1.3 and QUIC. Within a year or two, running DNS over TLS may very well become the rule rather than the experimental exception.

FreeBSD and CVE-2015-7547

As you have probably heard by now, a buffer overflow was recently discovered in GNU libc’s resolver code which can allow a malicious DNS server to inject code into a vulnerable client. This was announced yesterday as CVE-2015-7547. The best sources of information on the bug are currently Google’s Online Security Blog and Carlos O’Donnell’s in-depth analysis.

Naturally, people have started asking whether FreeBSD is affected. The FreeBSD Security Officer has not yet released an official statement, but in the meantime, here is a brief look at the issue as far as FreeBSD is concerned.

Continue reading “FreeBSD and CVE-2015-7547” »

DNS improvements in FreeBSD 11

Erwin Lansing just posted a summary of the DNS session at the FreeBSD DevSummit that was held in conjunction with BSDCan 2014 in May. It gives a good overview of the current state of affairs, including known bugs and plans for the future.

I’ve been working on some of these issues recently (in between $dayjob and other projects). I fixed two issues in the last 48 hours, and am working on two more.

Continue reading “DNS improvements in FreeBSD 11” »

VerifyHostKeyDNS

The Internet Society likes my work. I aim to please…

One of the things I did in the process of importing LDNS and Unbound into FreeBSD 10 was to change the default value for VerifyHostKeyDNS from “no” to “yes” in our OpenSSH when compiled with LDNS support (which can be turned off by adding WITHOUT_LDNS=YES to /etc/src.conf before buildworld).

The announcement the ISOC blog post refers to briefly explains my reasons for doing so:

I consider this a lesser evil than “ask” (aka “train the user to type ‘yes’ and hit enter”) and “no” (aka “train the user to type ‘yes’ and hit enter without even the benefit of a second opinion”).

There were objections to this (which I’m too lazy to dig up and quote) along the lines of:

  • Shouldn’t OpenSSH tell you that it found and used an SSHFP record?
  • Shouldn’t known_hosts entries take precedence over SSHFP records?
  • Shouldn’t OpenSSH store the key in known_hosts after verifying it against an SSHFP record?

The answer to all of the above is “yes, but…”

Here is how host key verification should work, ideally:

  1. Obtain host key from server
  2. Gather cached host keys from various sources (known_hosts, SSHFP, LDAP…)
  3. If we found one or more cached keys:
    1. Check for and warn about inconsistencies between these sources
    2. Check for and warn about inconsistencies between the cached key and what the server sent us
    3. If we got a match from a trusted source, continue connecting
    4. Inform the user of any matches from untrusted sources
  4. Display the key’s fingerprint
  5. Ask the user whether to:
    1. Store the server’s key for future reference and continue connecting
    2. Continue connecting without storing the key
    3. Disconnect

The only configuration required here is a list of trusted and untrusted sources, the difference being that a match or mismatch from a trusted source is normative while a match or mismatch from an untrusted source is merely informative.

Unfortunately, in OpenSSH, SSHFP support seems to have been grafted onto the existing logic rather than integrated into it. Here’s how it actually works:

  1. Obtain host key from server
  2. If VerifyHostKeyDNS is “yes” or “ask”, look for SSHFP records in DNS
  3. If an SSHFP record was found:
    1. If it matches the server’s key:
      1. If it has a valid DNSSEC signature and VerifyHostKeyDNS is “yes”, continue connecting
      2. Otherwise, set a flag to indicate that a matching SSHFP record was found
    2. Otherwise, warn about the mismatch
  4. Look for cached keys in the user and system host key files
  5. If we got a match from the host key files, continue connecting
  6. If we did not find anything in the host key files:
    1. If we found a matching SSHFP record, tell the user
    2. Ask the user whether to:
      1. Store the server’s key for future reference and continue connecting
      2. Disconnect
  7. If we found a matching revoked key in the host key files, warn the user and terminate
  8. If we found a different key in the host key files, warn the user and terminate

Part of the problem is that at the point where we tell the user that we found a matching SSHFP record, we no longer know whether it was signed. By switching the default for VerifyHostKeyDNS to “yes”, I’m basically saying that I trust DNSSEC more than I trust the average user’s ability to understand the information they’re given and make an informed decision.

DNS again: a clarification

There are a few points I’d like to clarify regarding my previous post about DNS in FreeBSD 10.

Some people were very quick to latch on to it and claim that “FreeBSD announced that Unbound and LNDS will replace BIND as the system’s DNS resolver” or words to that effect. This is, at best, a misunderstanding.

First of all: this is my personal blog. I speak only for myself, not for the FreeBSD project. I participated in the discussions and decision-making and did most of the work related to the switch, but I am neither a leader of nor a spokesperson for the project. As the current Security Officer, I sometimes speak on behalf of the project in security matters, but this is not one of those times. If this had been an official announcement, it would have been posted on the project’s website and / or on the freebsd-announce mailing list, not on my blog (or anybody else’s).

Second: BIND is a very mature, complex and versatile piece of software which implements pretty much every aspect of the DNS protocol and related standards, and is at the forefront of its field. It is developed and maintained by the Internet Systems Consortium, which is a major actor in the development and standardization of the DNS protocol. If you need an authoritative nameserver, or a caching resolver for a large and complex network, BIND is the natural choice. I use it myself, both privately and at work (note: I do not speak for the University of Oslo either). Most of the root servers run BIND. Unbound, on the other hand, is primarily a caching (recursing or forwarding) resolver. Although it has limited support for local zones (serving A, AAAA and PTR records only), which is mostly useful for overlaying information about machines on your RFC1918 SOHO network on top of the data served by a “real” nameserver, it is not capable of running as a full-fledged authoritative nameserver.

Third: due to its key role in Internet infrastructure, BIND is one of the most intensely scrutinized pieces of software. A tiny flaw in BIND can have major consequences for the Internet as a whole. The number and frequency of BIND-related security advisories are more a reflection of its importance than of its quality. Frankly, if you want to talk about code quality and BIND vs LDNS / Unbound… let’s just say that people who live in glass houses shouldn’t throw stones.

Fourth: FreeBSD has shipped with BIND for many years, but BIND was never the FreeBSD’s “system resolver” except in the loosest definition of the term. Most applications that need to perform DNS lookups use either gethostbyname(3) or, preferably, getaddrinfo(3), which are implemented entirely in libc (with code that goes back at least 25 years); I haven’t touched that code, and I don’t plan to. A few applications—notably host(1) and dig(1), which are actually part of BIND—need more fine-grained control over the queries they send and more details about the answers they receive, and use the BIND lightweight resolver library (lwres(3)); these applications have either been replaced by LDNS-based equivalents or deprecated. It is, of course, entirely possible to set up BIND as a local caching resolver; in fact, the default configuration we ship is perfectly suited to that purpose. It’s a little bit more work if you want it to forward rather than recurse—especially on a laptop or a broadband connection without a fixed IP, because you have to set up the resolvconf(8) magic yourself—but it’s not rocket surgery.

Fifth: a major part of the decision to remove BIND, which I stupidly forgot to mention, is that BIND 10 has been completely rewritten in C++ and Python. Importing Python into the base system is out of the question, so we would have been forced to switch sooner or later: at the earliest when users started complaining that we shipped an outdated version, and at the latest when the ISC discontinued BIND 9 entirely.

Sixth: Unbound is not a long-term solution. We needed a caching resolver for FreeBSD 10 and decided to use Unbound because it’s fairly mature and we know it well, but it is a stopgap measure to address the DNSSEC issue while we work on a long-term solution. For FreeBSD 11, we see DNS as only one of several services provided by the Capsicum service daemon called Casper; no decision has yet been made as to which validating resolver library Casper will use as its back-end. In any case, we will continue to provide both authoritative nameserver daemons and caching resolver daemons, such as BIND, NSD, Unbound DNSMasq etc. through the ports system, which can provide better support, access to newer versions, and faster updates than we can in the base system.

Finally, I should add that the ISC has supported the FreeBSD project for many years, both directly and indirectly. Although I haven’t been directly involved in that part of the project, I’m very grateful for their contribution and bear no ill will against them, and I was very unhappy to see my previous post misconstrued as an attack against BIND and the ISC.