FreeBSD – Page 3 – May Contain Traces of Bolts

VerifyHostKeyDNS

The Internet Society likes my work. I aim to please…

One of the things I did in the process of importing LDNS and Unbound into FreeBSD 10 was to change the default value for VerifyHostKeyDNS from “no” to “yes” in our OpenSSH when compiled with LDNS support (which can be turned off by adding WITHOUT_LDNS=YES to /etc/src.conf before buildworld).

The announcement the ISOC blog post refers to briefly explains my reasons for doing so:

I consider this a lesser evil than “ask” (aka “train the user to type ‘yes’ and hit enter”) and “no” (aka “train the user to type ‘yes’ and hit enter without even the benefit of a second opinion”).

There were objections to this (which I’m too lazy to dig up and quote) along the lines of:

Shouldn’t OpenSSH tell you that it found and used an SSHFP record?
Shouldn’t known_hosts entries take precedence over SSHFP records?
Shouldn’t OpenSSH store the key in known_hosts after verifying it against an SSHFP record?

The answer to all of the above is “yes, but…”

Here is how host key verification should work, ideally:

Obtain host key from server
Gather cached host keys from various sources (known_hosts, SSHFP, LDAP…)
If we found one or more cached keys:
1. Check for and warn about inconsistencies between these sources
2. Check for and warn about inconsistencies between the cached key and what the server sent us
3. If we got a match from a trusted source, continue connecting
4. Inform the user of any matches from untrusted sources
Display the key’s fingerprint
Ask the user whether to:
1. Store the server’s key for future reference and continue connecting
2. Continue connecting without storing the key
3. Disconnect

The only configuration required here is a list of trusted and untrusted sources, the difference being that a match or mismatch from a trusted source is normative while a match or mismatch from an untrusted source is merely informative.

Unfortunately, in OpenSSH, SSHFP support seems to have been grafted onto the existing logic rather than integrated into it. Here’s how it actually works:

Obtain host key from server
If VerifyHostKeyDNS is “yes” or “ask”, look for SSHFP records in DNS
If an SSHFP record was found:
1. If it matches the server’s key:
  1. If it has a valid DNSSEC signature and VerifyHostKeyDNS is “yes”, continue connecting
  2. Otherwise, set a flag to indicate that a matching SSHFP record was found
2. Otherwise, warn about the mismatch
Look for cached keys in the user and system host key files
If we got a match from the host key files, continue connecting
If we did not find anything in the host key files:
1. If we found a matching SSHFP record, tell the user
2. Ask the user whether to:
  1. Store the server’s key for future reference and continue connecting
  2. Disconnect
If we found a matching revoked key in the host key files, warn the user and terminate
If we found a different key in the host key files, warn the user and terminate

Part of the problem is that at the point where we tell the user that we found a matching SSHFP record, we no longer know whether it was signed. By switching the default for VerifyHostKeyDNS to “yes”, I’m basically saying that I trust DNSSEC more than I trust the average user’s ability to understand the information they’re given and make an informed decision.

DNS again: a clarification

There are a few points I’d like to clarify regarding my previous post about DNS in FreeBSD 10.

Some people were very quick to latch on to it and claim that “FreeBSD announced that Unbound and LNDS will replace BIND as the system’s DNS resolver” or words to that effect. This is, at best, a misunderstanding.

First of all: this is my personal blog. I speak only for myself, not for the FreeBSD project. I participated in the discussions and decision-making and did most of the work related to the switch, but I am neither a leader of nor a spokesperson for the project. As the current Security Officer, I sometimes speak on behalf of the project in security matters, but this is not one of those times. If this had been an official announcement, it would have been posted on the project’s website and / or on the freebsd-announce mailing list, not on my blog (or anybody else’s).

Second: BIND is a very mature, complex and versatile piece of software which implements pretty much every aspect of the DNS protocol and related standards, and is at the forefront of its field. It is developed and maintained by the Internet Systems Consortium, which is a major actor in the development and standardization of the DNS protocol. If you need an authoritative nameserver, or a caching resolver for a large and complex network, BIND is the natural choice. I use it myself, both privately and at work (note: I do not speak for the University of Oslo either). Most of the root servers run BIND. Unbound, on the other hand, is primarily a caching (recursing or forwarding) resolver. Although it has limited support for local zones (serving A, AAAA and PTR records only), which is mostly useful for overlaying information about machines on your RFC1918 SOHO network on top of the data served by a “real” nameserver, it is not capable of running as a full-fledged authoritative nameserver.

Third: due to its key role in Internet infrastructure, BIND is one of the most intensely scrutinized pieces of software. A tiny flaw in BIND can have major consequences for the Internet as a whole. The number and frequency of BIND-related security advisories are more a reflection of its importance than of its quality. Frankly, if you want to talk about code quality and BIND vs LDNS / Unbound… let’s just say that people who live in glass houses shouldn’t throw stones.

Fourth: FreeBSD has shipped with BIND for many years, but BIND was never the FreeBSD’s “system resolver” except in the loosest definition of the term. Most applications that need to perform DNS lookups use either gethostbyname(3) or, preferably, getaddrinfo(3), which are implemented entirely in libc (with code that goes back at least 25 years); I haven’t touched that code, and I don’t plan to. A few applications—notably host(1) and dig(1), which are actually part of BIND—need more fine-grained control over the queries they send and more details about the answers they receive, and use the BIND lightweight resolver library (lwres(3)); these applications have either been replaced by LDNS-based equivalents or deprecated. It is, of course, entirely possible to set up BIND as a local caching resolver; in fact, the default configuration we ship is perfectly suited to that purpose. It’s a little bit more work if you want it to forward rather than recurse—especially on a laptop or a broadband connection without a fixed IP, because you have to set up the resolvconf(8) magic yourself—but it’s not rocket surgery.

Fifth: a major part of the decision to remove BIND, which I stupidly forgot to mention, is that BIND 10 has been completely rewritten in C++ and Python. Importing Python into the base system is out of the question, so we would have been forced to switch sooner or later: at the earliest when users started complaining that we shipped an outdated version, and at the latest when the ISC discontinued BIND 9 entirely.

Sixth: Unbound is not a long-term solution. We needed a caching resolver for FreeBSD 10 and decided to use Unbound because it’s fairly mature and we know it well, but it is a stopgap measure to address the DNSSEC issue while we work on a long-term solution. For FreeBSD 11, we see DNS as only one of several services provided by the Capsicum service daemon called Casper; no decision has yet been made as to which validating resolver library Casper will use as its back-end. In any case, we will continue to provide both authoritative nameserver daemons and caching resolver daemons, such as BIND, NSD, Unbound DNSMasq etc. through the ports system, which can provide better support, access to newer versions, and faster updates than we can in the base system.

Finally, I should add that the ISC has supported the FreeBSD project for many years, both directly and indirectly. Although I haven’t been directly involved in that part of the project, I’m very grateful for their contribution and bear no ill will against them, and I was very unhappy to see my previous post misconstrued as an attack against BIND and the ISC.

DNS in FreeBSD 10

Yesterday, I wrote about the local caching resolver we now have in FreeBSD 10. I’ve fielded quite a few questions about it (in email and on IRC), and I realized that although this has been discussed and planned for a long time, most people outside the 50 or so developers who attended one or both of the last two Cambridge summits (201208 and 201308) were not aware of it, and may not understand the motivation.

There are two parts to this. The first is that BIND is a support headache with frequent security advisories and a lifecycle that aligns poorly with our release schedule, so we end up having to support FreeBSD releases containing a discontinued version of BIND. The second part is the rapidly increasing adoption of DNSSEC, which requires a caching DNSSEC-aware resolver both for performance reasons (DNSSEC validation is time-consuming) and to avoid having to implement DNSSEC validation in the libc resolver.

We could have solved the DNSSEC issue by configuring BIND as a local caching resolver, but for the reasons mentioned above, we really want to remove BIND from the base system; hence the adoption of a lightweight caching resolver. An additional benefit of importing LDNS (which is a prerequisite for Unbound) is that OpenSSH can now validate SSHFP records.

Note that the dns/unbound port is not going away, and that users who want to run Unbound as a caching resolver for an entire network rather than just a single machine have the option of either moving their configuration into /var/unbound/unbound.conf, or running the base and port versions side-by-side. This should not be a problem as long as the port version doesn’t try to listen on 127.0.0.1 or ::1.

I’d like to add that since my previous post on the subject, and with the help of readers, developers and users, I have identified and corrected several issues with the initial commit

/etc/unbound is now a symlink to /var/unbound. My original intention was to have the configuration files in /etc/unbound and the root anchor, unbound-control keys etc. in /var/unbound, but the daemon needs to access both locations at run-time, not just on start-up, so they must all be inside the chroot. Running the daemon un-chrooted is, of course, out of the question.
The init script ordering has been amended so the local_unbound service now starts before most (hopefully all) services that need functioning DNS.
resolvconf(8) is now blocked from updating /etc/resolv.conf to avoid failing over from the DNSSEC-aware local resolver to a potentially non-DNSSEC-aware remote resolver in the event of a request returning an invalid record.
The configure command line and date / time are no longer included in the binary.

Finally, I just flipped the switch so that BIND is now disabled by default and the LDNS utilities are enabled. The BIND_UTILS and LDNS_UTILS build options are mutually exclusive; in hindsight, I should probably have built and installed the new host(1) as ldns-host(1) so both options could have been enabled at the same time. We don’t yet have a dig(1) wrapper for drill(1), so host(1) is the only actual conflict.

Local caching resolver in FreeBSD 10

As of a few hours ago, all it takes to set up a local caching resolver in FreeBSD 10 is:

# echo local_unbound_enable=yes >>/etc/rc.conf
# service local_unbound start

Yes, it really is that simple—and it works fine with DHCP, too. Hold my beer and watch this:

# pgrep -lf dhclient
1316 dhclient: vtnet0
1265 dhclient: vtnet0 [priv]
# cat /etc/resolv.conf
# Generated by resolvconf
search example.com
nameserver 192.0.2.53
# time host www.freebsd.org
www.freebsd.org is an alias for wfe0.ysv.freebsd.org.
wfe0.ysv.freebsd.org has address 8.8.178.110
wfe0.ysv.freebsd.org has IPv6 address 2001:1900:2254:206a::50:0
wfe0.ysv.freebsd.org mail is handled by 0 .
        0.02 real         0.00 user         0.01 sys

As you can see, we’re running DHCP on a VirtIO network interface. Let’s work our magic:

# echo local_unbound_enable=yes >>/etc/rc.conf
# service local_unbound start
Performing initial setup.
Extracting forwarders from /etc/resolv.conf.
/var/unbound/forward.conf created
/var/unbound/unbound.conf created
/etc/resolvconf.conf created
original /etc/resolv.conf saved as /etc/resolv.conf.20130923.075319
Starting local_unbound.

And presto:

# pgrep -lf unbound
3799 /usr/sbin/unbound -c/var/unbound/unbound.conf
# cat /var/unbound/unbound.conf 
# Generated by local-unbound-setup
server:
        username: unbound
        directory: /var/unbound
        chroot: /var/unbound
        pidfile: /var/run/local_unbound.pid
        auto-trust-anchor-file: /var/unbound/root.key

include: /var/unbound/forward.conf
# cat /var/unbound/forward.conf
# Generated by local-unbound-setup
forward-zone:
        name: .
        forward-addr: 192.0.2.53
# cat /etc/resolv.conf
# Generated by resolvconf
search example.com
# nameserver 192.0.2.53

nameserver 127.0.0.1
options edns0

We can see the cache at work; the first request takes significantly longer than before, but the second is served from cache:

# time host www.freebsd.org
www.freebsd.org is an alias for wfe0.ysv.freebsd.org.
wfe0.ysv.freebsd.org has address 8.8.178.110
wfe0.ysv.freebsd.org has IPv6 address 2001:1900:2254:206a::50:0
wfe0.ysv.freebsd.org mail is handled by 0 .
        0.07 real         0.01 user         0.00 sys
# time host www.freebsd.org
www.freebsd.org is an alias for wfe0.ysv.freebsd.org.
wfe0.ysv.freebsd.org has address 8.8.178.110
wfe0.ysv.freebsd.org has IPv6 address 2001:1900:2254:206a::50:0
wfe0.ysv.freebsd.org mail is handled by 0 .
        0.01 real         0.00 user         0.00 sys

Finally, let’s see how this interacts with DHCP:

# resolvconf -u
# cat /etc/resolv.conf
# Generated by resolvconf
search example.com
nameserver 127.0.0.1
nameserver 192.0.2.53
options edns0

# cat /var/unbound/forward.conf 
# Generated by resolvconf

forward-zone:
        name: "example.com"
        forward-addr: 192.0.2.53

forward-zone:
        name: "."
        forward-addr: 192.0.2.53

Note that resolvconf(8) re-added the 192.0.2.53 entry. It doesn’t really matter, as long as 127.0.0.1 comes first.

[ETA: it does matter—see Jakob Schlyter’s comment below and my reply.]

[ETA: see my followup about the motivation for importing Unbound.]

Growing a VirtualBox disk with ZFS on it

I have a VirtualBox VM on a Windows host with a 32 GB disk. That disk is partitioned with GPT and has four partitions: a boot partition, a swap partition, a smallish UFS root partition, and a ZFS partition. I need more space in the latter, so let’s grow it.

Continue reading “Growing a VirtualBox disk with ZFS on it”