FreeBSD Email Server - Part 6: System Updates

Four years have past, and my FreeBSD email server has keps on running without any problems. However, some people on IRC have recently been nagging me to support TLSv1.3 on my mailserver. Since the installation was done 4 years ago, it didn’t do 1.3 yet, just 1.2. I set out to do a relatively simple system update, which didn’t go as smooth as I had hoped. This tutorial post should help you avoid the mistakes I made, so your updates will go smooth.

Preparations

Before we do anything wild, let’s do the obvious first step: backups. Since this is a FreeBSD server, it uses glorious ZFS as the filesystem, which allows us to make use of snapshots. Which subvolumes to make snapshots off depends on your particular setup. In my case, my actual email data is stored on zroot/srv, and all the services and their configurations are in zroot/usr/local. My database’s data is stored on zroot/postgres/data96. Additionally, I want to make a snapshot of zroot/usr/ports.

zfs snapshot -r zroot/srv@`date +%Y%m%d%H%M%S`-11.0-final
zfs snapshot -r zroot/usr/local@`date +%Y%m%d%H%M%S`-11.0-final
zfs snapshot -r zroot/postgres@`date +%Y%m%d%H%M%S`-11.0-final
zfs snapshot -r zroot/usr/ports@`date +%Y%m%d%H%M%S`-11.0-final

This will make a snapshot of each of these locations, for easy restoration in case any problems arise. You can list all your snapshots with zfs list -t snapshot.

Your server is most likely hosted at a provider, not in your home. This means you won’t be able to just physically access it and retrieve the harddrive if things go really bad. You might not be able to boot single-user mode either. Because of this, you might not be able to restore the snapshots if things go really bad. In this case, you should also make a local copy of the important data.

The services and their configuration can be recreated, just follow the earlier parts of this series again. The email data, however, cannot. This is the data in /srv/mail. You can make a local copy of all this data using rsync.

rsync -av example.org:/srv/mail/ ~/mail-backup

There’s one more thing to do, which I learned the hard way. Set your login shell to a simple one, provided by the base system. The obvious choice is /bin/sh, but some people may wrongly prefer /bin/tcsh as well. During a major version update, the ABI changes, which will temporarily break most of the user-installed packages, including your shell.

chsh

Updating the Base System

With the preparations in place in case things get royally screwed up, the actual updates can begin. FreeBSD has a dedicated program to handle updating the base system, freebsd-update. First off, fetch any updates, and make sure all the updates for your current version are applied.

freebsd-update fetch install

Afterwards, set the new system version you want to update to. In my case, this is 12.1-RELEASE, but if you’re reading this in the future, you most certainly want a newer version.

freebsd-update -r 12.1-RELEASE upgrade

This command will ask you to review the changes and confirm them as well. It should generally be fine, but this is your last chance to make any backups or perform other actions to secure your data! If you’re ready to continue, install the updates to the machine.

freebsd-update install

At this point, your kernel has been updated. Next you must reboot to start using the new kernel.

reboot

Once the system is back online, you can continue installing the rest of the updates.

freebsd-update install

When this command finishes, the base system has been updated and should be ready for use. Next up is updating all the software you installed manually.

Updating User-Installed Packages

Unlike GNU+Linux distributions, FreeBSD has a clear distinction between the base system and user installed software. The base system has now been updated, but everything installed through pkg or ports is still at the old version. If you performed a major version upgrade (say, FreeBSD 11.x to 12.x), the ABI has changed and few, if any, of the user-installed packages still work.

Binary Packages using pkg

Binary packages are the most common packages used. These are the packages installed through pkg. Currently, pkg itself doesn’t even work. Luckily, FreeBSD has pkg-static, which is a statically compiled version of pkg intended to fix this very problem. Let’s fix up pkg itself first.

pkg-static install -f pkg

That will make pkg itself work again. Now you can use pkg to update package information, and upgrade all packages to a version that works under this FreeBSD version.

pkg update
pkg upgrade

PostgreSQL

A particular package that was installed through pkg, PostgreSQL, just got updated to the latest version. On FreeBSD, the data directory used by PostgreSQL is dependent on the version you’re running. If you try to list databases now, you’ll notice that the mail database used throughout the tutorial is gone. The data directory is still there, so you could downgrade PostgreSQL again, restart the database, run a pgdump, upgrade, restart and import. However, I find it much cleaner to use FreeBSD jails to solve this issue.

I generally put my jails in a ZFS subvolume, so let’s create one of those first.

zfs create -o mountpoint=/usr/jails zroot/jails
zfs create zroot/jails/postgres96

This will create a new subvolume at /usr/jails/postgres96. Using bsdinstall, a clean FreeBSD installation usable by the jail can be set up here. This command will give you some popups you may remember from installing FreeBSD initially. This time, you can uncheck all boxes, to get the most minimal system.

bsdinstall jail /usr/jails/postgres96

When bsdinstall finishes, you can configure the jail. This is done in /etc/jail.conf. If this file doesn’t exist, you can create it. Make sure the following configuration block is written to the file.

postgres96 {
    # Init information
    exec.start = "/bin/sh /etc/rc";
    exec.stop  = "/bin/sh /etc/rc.shutdown";
    exec.clean;

    # Set the root path of the jail
    path = "/usr/jails/$name";

    # Mount /dev
    mount.devfs;

    # Set network information
    host.hostname = $name;
    ip4.addr = "lo0|127.1.1.1/32";
    ip6.addr = "lo0|fd00:1:1:1::1/64";

    # Required for PostgreSQL to function
    allow.raw_sockets;
    allow.sysvipc;
}

Now you can start up the jail, so it can be used.

service jail onestart postgres96

Using the host system’s pkg, you can install PostgreSQL into the jail.

pkg -c /usr/jails/postgres96 install postgresql96-server

Now you just need to make the data directory available to the jail, which you can most easily do using nullfs.

mount -t nullfs /var/db/postgres/data96 /usr/jails/postgres96/var/db/postgres/data96

Now everything should be ready for use inside the jail. Let’s head on in using jexec.

jexec postgres96

Once inside the jail, you can start the PostgreSQL service, and dump the mail database.

service postgresql onestart
su - postgres
pg_dump mail > ~/mail.sql

This will write the dump to /usr/jails/postgres96/var/db/postgres/mail.sql on the host system. You can leave the jail and close it down again.

exit
exit
service jail onestop postgres96

This dump can be imported in your updated PostgreSQL on the host system. Connect to the database first.

su - postgres
psql

Then, recreate the user, database and import the data from the dump.

CREATE USER postfix WITH PASSWORD 'incredibly-secret!';
CREATE DATABASE mail WITH OWNER postfix;
\c mail
\i /usr/jails/postgres96/var/db/postgres/mail.sql
\q

The mail database is now back, and ready for use!

Packages from Ports

With all the binary packages out of the way, it’s time to update packages from ports. While it is very possible to just go to each port’s directory and manually update each one individually, I opted to use portupgrade. This will need manual installation, but afterwards, we can rely on portupgrade to do the rest. Before doing anything with the ports collection, it should be updated, which is done using portsnap.

portsnap fetch extract

Once this is done, you can go to the portupgrade directory and install it.

cd /usr/ports/ports-mgmt/portupgrade
make install clean

Now, to upgrade all other ports.

portupgrade -a

Be sure to double-check the compilation options that you are prompted about! If you’re missing a certain option, you may miss an important feature that is required for your mailserver to work appropriately. This can be easily fixed by recompiling, but a few seconds checking now can save you an hour figuring it out later!

Tidying Up

Now that all user-installed software has been updated too, it’s time to finalize the update by running freebsd-update for a final time.

freebsd-update install

You can return to your favourite shell again.

chsh

And you can clean up the ports directories to get some wasted space back.

portsclean -C

I would suggest making a new snapshot as well, now that you’re on a relatively clean and stable state.

zfs snapshot -r zroot/srv@`date +%Y%m%d%H%M%S`-12.1-clean
zfs snapshot -r zroot/usr/local@`date +%Y%m%d%H%M%S`-12.1-clean
zfs snapshot -r zroot/postgres@`date +%Y%m%d%H%M%S`-12.1-clean
zfs snapshot -r zroot/usr/ports@`date +%Y%m%d%H%M%S`-12.1-clean

And that concludes your system update. Your mailserver is ready to be neglected for years again!