FreeBSD Disk Operations

The FreeBSD install described in Portable ZFS Pool gives you a highly robust system. But eventually a disk will fail, and you will need to replace it. This page is my cheatsheet to help me remember how things work. (If you use anything on this page, don’t just cut and paste commands, think about what they mean and double check device names. The examples are just examples.)

Disk and partition naming convention

Every partition in my environment has a unique GPT label. This is to prevent accidents that may happen when two disks with identically named partitions are plugged in at the same time.

I put physical stickers on disks, and write the disk’s name. Then I use the disk’s name in its partition names. I can tell a partition’s physical disk and function just by looking at the name.

My convention for GPT partition names is disk-typeN-functionM.

disk-type is usb, sata, and so on. N is just an incrementing integer for every disk I use. So a disk with a sticker “USB10” on it will have partitions that start with usb10-. For example, usb10-efi0. (I also write the hostname on the sticker, so disks for the same hosts can easily be paired and I don’t get them mixed up.)

M is usually 0, but can be something else if you have more than one of the same type of partition on the disk.

This reference says GPT partition labels can be up to 72 characters long. https://wiki.archlinux.org/title/Persistent_block_device_naming

Geom mirrors are used for swap, and the names of these mirrors live under /dev/mirror/. The names of geom mirrors are also be unique within my environment.

My convention for mirror names is host-functionN. For example, fs0-swap0. This convention prevents confusion if a disk is plugged in to another host.

This reference says the max length of a mirror name is 15: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254146

Layout of system disks

When you install FreeBSD using the “BIOS+UEFI” option and GPT partitions, your disk ends up split like this:

Section Partition Type Size Index Used By
pmbr n/a 512 bytes in sector 0 BIOS
GPT partition table n/a 17k EFI
ESP efi 260M 1 EFI
boot freebsd-boot 512k 2 BIOS
swap freebsd-swap 2G default 3 kernel
zfs freebsd-zfs rest of disk 4 kernel

You don’t see the PMBR or GPT partition table directly, and there can be free space between partitions to ensure alignment on 4k or 1M boundaries.

Use gpart show to see a disk’s partitioning.

Fixing GPT labels and swap device names

The labels created by the standard FreeBSD install do not conform to the naming convention above, and so there is the potential for clashes if disks from different installs end up plugged into the same system. This is how you can fix the labels on a standard install.

First check /etc/fstab for device names. For example, the standard install wants to mount the ESP on /boot/efi, but it uses a device name instead of a GPT label. Comment this out of /etc/fstab.

For swap, use the GPT name if the system has a single disk, or use a gmirror name if the system is using mirrored swap. For single disks, change the swap device to something like this:

/dev/gpt/sata0-swap0.eli

For mirrored swap, change the swap device to something like this:

/dev/mirror/fs0-swap0.eli

(The .eli suffix enables encrypted swap.)

Now boot into the install CD/USB and choose Live CD. Run the following, setting d and n according to your setup:

d=ada0
n=sata0

gpart modify -l $n-efi0 -i 1 $d
gpart modify -l $n-boot0 -i 2 $d
gpart modify -l $n-swap0 -i 3 $d
gpart modify -l $n-zfs0 -i 4 $d

If your system was installed with mirroring already set up, you’ll need to modify the labels on the second drive as well. Repeat the above commands using d=ada1 and n=sata1 (for example).

Update the zroot pool’s vdev device names:

mkdir /tmp/mnt
zpool import -R /tmp/mnt -d /dev/gpt zroot
zpool export zroot

Then reboot into single user mode.

If swap is already mirrored, you’ll have to rebuild it with a new name that fits the convention:

gmirror destroy swap0
gmirror label fs0-swap0 gpt/sata0-swap0 gpt/sata1-swap0

Now if you exit and go to multi-user mode, ZFS will use GPT names in its vdevs, and your /etc/fstab will pick up the correct swap device.

exit

Partitioning

Manual partitioning

When you replace a disk (or add a mirror), you’ll have to partition the new disk the same as the original disk. Below is the way the installer does it. If you are replacing a disk that was built by the FreeBSD installer, follow these steps. If not, use gpart show to decipher the existing partitioning.

d=ada1
n=sata0

# destroy the existing partitioning
gpart destroy -F $d

# create GPT partition scheme
gpart create -s GPT $d

# create the ESP
gpart add -a 4k -t efi -s 260M -l $n-efi0 $d

# create FreeBSD boot partition
gpart add -a 4k -t freebsd-boot -s 512k -l $n-boot0 $d

# create 2G swap partition
gpart add -a 1M -t freebsd-swap -s 2G -l $n-swap0 $d

# use the rest for ZFS
gpart add -a 1M -t freebsd-zfs -l $n-zfs0 $d

Note that partitioning does not install boot blocks. See below for that.

Duplicating a partition table

You can duplicate the partition table from one disk to another. This only makes sense when both disks are exactly the same size.

In this example we copy the partition table from ada0 to ada1. The -F option means force. gpart restore will refuse to do anything if there is already a valid partition table on the drive.

gpart backup ada0 | gpart restore -F ada1

In that example, the new partitions on ada1 do not get GPT labels. You could use gpart restore -l, but then the labels on the new disk would be identical to the labels on the original disk. Identical labels will confuse you and the operating system.

If the labels on ada0 use the partition naming convention described above, you can automatically create the appropriate labels on ada1. For example, if the labels on ada0 start with “sata5” and the labels on ada1 should start with “sata6”, you can do this:

gpart backup ada0 | sed -e 's/sata5-/sata6-/' | gpart restore -lF ada1

Installing boot blocks

Once the partitions are in place you can install the pieces needed to make the disk bootable. There are three things that must be installed:

  • Protective MBR in sector 0
  • gptzfsboot in the freebsd-boot partition
  • The EFI loader in the ESP

Boot blocks must be installed on all potentially bootable disks because boot blocks are not included in these mirroring schemes.

Protective MBR

The MBR is sector 0 of a boot disk and is used by BIOS systems.

Install it like this:

gpart bootcode -b /boot/pmbr ada1

/boot/pmbr allows a BIOS system to boot from a GPT-partitioned disk. It has enough code to locate a GPT freebsd-boot partition and run its boot code. See boot(8).

gptzfsboot

The freebsd-boot partition is not formatted with any filesystem. Its contents are simply /boot/gptzfsboot copied byte for byte.

Install like this:

gpart bootcode -p /boot/gptzfsboot -i 2 ada1

-i 2 means partition index 2, the freebsd-boot partition in the table above.

The first 512 bytes of gptzfsboot are loaded and run by pmbr. The code in this first block then loads the rest of the partition and runs that.

The gptzfsboot program locates the ZFS boot pool and runs /boot/loader. See gptzfsboot(8).

EFI loader

The loaders in the ESP partition are used by UEFI systems. This partition has a normal FAT32 filesystem, and is located by UEFI by its filesystem type efi.

Run the following to setup the ESP with the FreeBSD loader:

newfs_msdos /dev/gpt/sata0-efi0
mount -t msdos /dev/gpt/sata0-efi0 /boot/efi

mkdir -p /boot/efi/efi/freebsd
cp /boot/loader.efi /boot/efi/efi/freebsd/loader.efi

mkdir -p /boot/efi/efi/boot
cp /boot/loader.efi /boot/efi/efi/boot/bootx64.efi

umount /boot/efi

Mirrored swap

Swap space is mirrored with geom mirroring and optionally geli encryption. Swap should be mirrored because it prevents a single disk failure from crashing the system. (Swap is memory, and a disk crash is equivalent to failing a memory stick at runtime.)

You should encrypt swap so passwords and other sensitive information don’t end up on disk.

Creating a mirrored swap area

If your current swap area is not mirrored, this is how to set up a mirror and make sure the mirror is mentioned in /etc/fstab.

These steps assume your partitions are labelled according to the convention above (eg the swap partitions are named something like “sata0-swap0” and “sata1-swap1”). And in this example, your host is “fs0”.

Ensure the gmirror module is loaded into the kernel at boot

Make sure this is in /boot/loader.conf for future boots:

geom_mirror_load="YES"

And load the module into the running kernel:

gmirror load

Turn off swapping

If the system is not using much swap, you can just run swapoff to temporarily turn off swapping. Go to single user mode if you can’t turn off swap.

swapoff /dev/ada0p3

Create the mirror

gmirror label fs0-swap0 gpt/sata0-swap0 gpt/sata1-swap0

Fixup /etc/fstab

/etc/fstab must be changed to use the mirror for swap. Do something like this:

sed -i .org -e 's|/dev/ada0p3|/dev/mirror/fs0-swap0|' /etc/fstab

Use your old and new swap device names in the sed substitution commands.

If you want to encrypt swap, ensure the swap device name in /etc/fstab ends in .eli.

Turn swapping back on

Now you can reboot or turn swapping back on:

swapon /dev/mirror/fs0-swap0.eli

Adding a swap mirror

This step is for when you are already using a gmirror for swap and you want to add another component to the mirror.

Clear metadata from the new component

gmirror clear gpt/sata2-swap0

Add the new component to the mirror:

gmirror insert fs0-swap0 gpt/sata2-swap0

Use gmirror status to show resyncing progress. When the mirror is fully synced you can use the steps below to remove or forget a failed mirror.

Removing a swap mirror

You can remove a component from a mirror as long as the mirror always has at least one good component.

gmirror remove fs0-swap0 gpt/sata1-swap0

If that fails because the drive is gone or has crashed and burned, use forget below.

Forgetting a failed mirror

If a disk has failed and gmirror remove doesn’t work, you can tell the mirror to just forget that component:

gmirror forget fs0-swap0 gpt/sata1-swap0

Mirrored ZFS

We don’t have to set up ZFS from scratch because it has already been set up by the operating system installer. The only operations we need to worry about are attaching and detaching components from the vdev.

Attaching a new device to a vdev

Use zpool status to determine the device name of one of the good components of the vdev. Then attach the new device to that good component. Here sata0-zfs0 is the good component, and sata2-zfs0 is the one you want to attach.

zpool attach zroot gpt/sata0-zfs0 gpt/sata2-zfs0

Use zpool status to monitor resilvering progress. When resilvering is complete, you can use the step below to remove a failed component.

Detaching a device from a vdev

You don’t really have to wait until resilvering is complete before you detach the failed device, as long as there is always one good copy of the data. But sometimes it is slightly better to leave the failed device until the new one is online. See Lucas for why.

zpool detach zroot gpt/sata1-zfs0

References