pkgbase: what can go wrong?

Yesterday I wrote about how I ended up manually converting a FreeBSD 15 server and the eight jails it hosts to packaged base manually. I dashed that post off in a bit of a hurry because I knew that if I put it off until later I would probably never get around to it. 20 or so hours later, here’s a followup with some caveats and some tips on how to ensure the conversion goes well.

First, I should point out that the procedure outlined in yesterday’s post has no guardrails and should not be attempted unless you are very comfortable with FreeBSD system administration and willing and able to fix things if they go awry.

Second, perhaps I should elaborate on why I (mostly) skipped 15.0. We (the FreeBSD project collectively) have been working on packaged base for over a decade. At some point we thought 14.0 was when it would finally land, but that did not pan out. Then it was going to be 15.0, and there was a massive push to get everything ready, both in base and in pkg… and we still failed: packaged base in FreeBSD 15 is officially a technology preview, and FreeBSD 16 is when we will finally transition. As a result, the pre-release churn on the stable/15 branch was much higher than for a normal major release and a number of things fell through the cracks (including some that had nothing to do with packaged base). The bottom line is that I did not feel that 15.0 was production-ready when it came out, but I worked hard to get most of the issues I was aware of sorted out for 15.1, and I feel much better about 15.1 than I did about 15.0.

Third, I left out a couple of steps for brevity. Most importantly, the systems I converted were all on ZFS (eight jails and their host, as previously mentioned) which allowed me to take a snapshot of the root filesystem (which in a standard FreeBSD install includes /var and /usr/local) so if something had gone wrong I could simply have reverted to that, or rebooted to the boot environment that freebsd-update created when it upgraded my system to 15.1.

My case is also atypical because I have my own package repositories, including for packaged base. I converted the host to packaged base using the official FreeBSD-base repository first, then switched to my own before running pkg upgrade -fy; I then converted the jails directly to my own repository without passing through FreeBSD-base. This is somewhat risky because my base packages are built with non-standard options and do not match FreeBSD distribution sets perfectly, but I know exactly where they differ and how to deal with it.

Assuming your system meets the prerequisites (recently updated binary install of 15.0 or 15.1), the worst thing that can happen is that you forget to install a set and end up with unregistered files (e.g. you had tests.txz installed on your system but left out FreeBSD-set-tests from your pkg install --register-only command line). This is easily remedied, even after the fact, by registering that set and then removing it:

# pkg install --register-only -y FreeBSD-set-tests
# pkg remove -fy FreeBSD-set-tests
# pkg autoremove -y

But watch out! FreeBSD-set-tests depends on FreeBSD-set-base which depends on FreeBSD-set-minimal and FreeBSD-set-optional, so if you had already trimmed your system down to e.g. FreeBSD-set-minimal-jail you will have to remove those extra sets again:

# pkg remove -fy FreeBSD-set-{base,minimal,optional}
# pkg autoremove -y

Since we used --register-only, one thing that can’t happen is that pkg replaces a modified configuration file with its stock version, saving the modified version with a .pkgsave extension. On the other hand, it is possible that pkg upgrade -fy fails to install a file because a directory of the same name exists. I’ve had that happen exactly once due to a discrepancy in /usr/src (I’m not sure why but freebsd-update(8) has never been entirely reliable when it came to patching the source tree). So you may want to run the following after converting:

# find -xs / -name '*.pkgnew'

If you get any hits, there will usually be an empty directory of the same name (without the .pkgnew suffix) which you can remove before manually renaming the file.

Another useful post-conversion command is:

# find -xs / \( -name var -prune \) -or -not -type d -print0 |
xargs -0 pkg which 2>&1 | grep 'not found' | less

This will give you a list of files on the root filesystem which pkg does not know about. You should be able to identify most of them (configuration files, the trust store, miscellaneous generated files such as /etc/spwd.db).

Finally, look for empty directories on the root filesystem:

# find -xs / -type d -empty

There will usually be a few in /usr/lib/debug and /usr/tests which you can remove if you did not choose to install debugging symbols or tests.

And my final piece of advice, if you want minimum hassle at the cost of having the machine do some pointless work that immediately gets reverted, just register all the sets, then remove the ones you don’t have or don’t want. You’ll get errors from autoremove about missing files, but that (and the few dozen extra seconds the process will take) is about the only downside.

pkgbasify

How to easily convert your FreeBSD 15 system to packaged base

I just spent the day catching up on FreeBSD email and events after a week of marshaling, and upgrading various systems to FreeBSD 15.1. Most of my various systems (servers, VMs, and jails) are still running 14.3 or 14.4 because, in my opinion, 15.0 shipped with too many unresolved warts, but now that 15.1 is out (albeit not yet announced) it is high time to not only upgrade everything to 15.1 but also make the switch to packaged base (which I was so far only using on one VM and one jail).

After a brief look at the FreeBSD Foundation’s pkgbasify script I came to the conclusion that I do not trust it to work correctly, so I developed my own method for converting a system from distribution sets to packaged base. Assuming a full install (including debug symbols, source tree, and tests) of FreeBSD 15.0-RELEASE or 15.1-RELEASE with pkg 2.7.5 recently updated to the latest patch level (if any), you can convert to packaged base with a single command:

# pkg install -r FreeBSD-base --register-only FreeBSD-set-{base,lib32,kernels}{,-dbg} FreeBSD-set-{src,tests}

Note 1: I’m assuming that you are using bash or zsh or some other shell that supports curly brace notation; FreeBSD’s /bin/sh does not.

Note 2: the pkg-install(8) manual page claims that you can abbreviate --register-only to -X, but you can’t, although that will be fixed in the next version.

There may be minor discrepancies between what the installer and freebsd-update(8) produced and what the packages contain, which you can usually fix with a forced upgrade:

# pkg upgrade -r FreeBSD-base -fy

If you do not have a full install, you should trim down the pkg install command line to only the sets that correspond to what you have; otherwise, the subsequent pkg upgrade will install what was missing. Particularly, you should not install FreeBSD-set-src if /usr/src is already managed by Git.

Once you’re done, you can start trimming down your system by removing sets you don’t need, then autoremoving their contents. For instance, if you’ve decided that you no longer need tests or debugging symbols:

# pkg remove -fg FreeBSD-set-\*-dbg FreeBSD-set-tests
# pkg autoremove

Note that package sets are marked vital so you must use -f when removing them.

Now is the time to enable the FreeBSD-base repository, by the way. Manually specifying it with -r FreeBSD-base temporarily enables it for the current command, but you’ll want it permanently enabled:

# mkdir -p /usr/local/etc/pkg/repos
# echo 'FreeBSD-base: { enabled: yes }' >/usr/local/etc/pkg/repos/FreeBSD-base.conf

Finally, if on bare metal or a VM, create a boot environment of your converted system:

# bectl create $(pkg query %v FreeBSD-runtime)

The only issue I’ve noticed after converting one bare-metal server and the eight jails it hosts is that pkg leaf does not take shared library dependencies into account:

# sudo pkg -j myjail leaf
FreeBSD-bluetooth-lib-15.1
FreeBSD-set-base-jail-15.1
FreeBSD-ufs-lib-15.1
FreeBSD-zfs-lib-15.1
pkg-2.7.5

Those three -lib packages provide libraries which are used by other base packages, but the dependencies are all implicit, so they show up as leaves despite not actually being removable.

In closing, I would like to thank Baptiste Daroussin and Lexi Winter for their suggestions while I was working out how I was going to do this, and to Emmanuel Vadot for implementing the --register-only option.

DNS over TLS in FreeBSD with Quad9

It has come to my attention that Quad9 have a blog post providing incorrect instructions for how to set up a FreeBSD system to use their service. I have attempted to get in touch with the author and get him to correct it but have received no response. So here, for the benefit of the Great Search Engine Gods, is the correct procedure; see my earlier post on the topic for more details on how it works.

# cat >/etc/rc.conf.d/local_unbound <<EOF
local_unbound_enable="yes"
local_unbound_tls="yes"
local_unbound_forwarders="9.9.9.9@853#dns.quad9.net 149.112.112.112@853#dns.quad9.net 2620:fe::fe@853#dns.quad9.net 2620:fe::9@853#dns.quad9.net"
EOF
# service local_unbound setup
# service local_unbound restart

No need to reboot.

Note that if you only have IPv4, you may experience slightly degraded performance unless you leave out the IPv6 addresses from the local_unbound_forwarders line (and vice versa in the unlikely scenario where you only have IPv6).

Automatic Let’s Encrypt certificates in Apache with mod_md

Since 2.4.30, Apache comes with experimental support for ACME certificates (Let’s Encrypt et al.) in the form of mod_md (short for “managed domains”). It’s kind of a pain but it’s still better than what I had before, i.e. a mess of shell and Perl scripts based on Crypt::LE, and if your use case is limited to Apache, it appears to be simpler than Certbot as well. Unfortunately for me, it’s not very well documented and I wasted a considerable amount of time figuring out how to use it. Fortunately for you, I then decided to blog about it so you don’t have to repeat my mistakes.

Edit: the author of mod_md, Stefan Eissing, got in touch and pointed me to his own documentation, which is far superior to the one available from Apache.

Continue reading “Automatic Let’s Encrypt certificates in Apache with mod_md”

DNS over TLS in FreeBSD 12

With the arrival of OpenSSL 1.1.1, an upgraded Unbound, and some changes to the setup and init scripts, FreeBSD 12, currently in beta, now supports DNS over TLS out of the box. We show how to set it up and discuss its advantages and disadvantages.

With the arrival of OpenSSL 1.1.1, an upgraded Unbound, and some changes to the setup and init scripts, FreeBSD 12.0, currently in beta, now supports DNS over TLS out of the box.

DNS over TLS is just what it sounds like: DNS over TCP, but wrapped in a TLS session. It encrypts your requests and the server’s replies, and optionally allows you to verify the identity of the server. The advantages are protection against eavesdropping and manipulation of your DNS traffic; the drawbacks are a slight performance degradation and potential firewall traversal issues, as it runs over a non-standard port (TCP port 853) which may be blocked on some networks. Let’s take a look at how to set it up.

Continue reading “DNS over TLS in FreeBSD 12”