Michael's Daemonic Doodles

...blogging bits of BSD

Setting up FreeBSD on ZFS raidz2 using mfsBSD

As I explained in an earlier post, we had issues with one of our backup servers. I decided to do a complete reinstall and ended up using an LSI SAS 9211 controller running in initiator-target mode (IT firmware), so all eight disks show up as plain HBAs and can be used to form a zpool. After considering my options I decided to go with raidz2, which will provide a good balance of storage space and safety.


Since I had great results using Martin Matuška's brilliant mfsBSD collection in the past, I also used it for setting up FreeBSD 9.0 on the backup host. Custom made versions of mfsBSD are really easy to create and we used a specially hardened custom distribution of mfsBSD for a PCI DSS compliant setup in the past. In this case the precompiled 9.0-RELEASE amd64 special edition, which can downloaded from the project's home page, works just fine.

Basic Installation

After booting mfsBSD 9.0-RELEASE amd64 special edition, login using the credentials root/mfsroot. Next, mount the CD containing installation data:

mount -t cd9660 /dev/cd0 /cdrom


The exact name of the device depends on the installation medium used.

Unfortunately at this point zfsinstall doesn't support raidz out of the box. This is easily fixed by modifying the script:

sed -i .old s/raidz/raidz2/g /root/bin/zfsinstall

Now zfsinstall can be run (this assumes that there is no existing ZFS container on the drives, drive names are /dev/da[0-8], 16GB of swap).

zfsinstall -t /cdrom/9.0-RELEASE-amd64.tar.xz -s 16G -r raidz2 \
-d da0 -d da1 -d da2 -d da3 -d da4 -d da5 -d da6 -d da7

Zfsinstall doesn't label the partitions, to make life easier it's best to manually label swap partitions ...

gpart modify -i 2 -l swap0 da0
gpart modify -i 2 -l swap1 da1
gpart modify -i 2 -l swap2 da2
gpart modify -i 2 -l swap3 da3
gpart modify -i 2 -l swap4 da4
gpart modify -i 2 -l swap5 da5
gpart modify -i 2 -l swap6 da6
gpart modify -i 2 -l swap7 da7

... and ZFS pool members:

gpart modify -i 3 -l disk0 da0
gpart modify -i 3 -l disk1 da1
gpart modify -i 3 -l disk2 da2
gpart modify -i 3 -l disk3 da3
gpart modify -i 3 -l disk4 da4
gpart modify -i 3 -l disk5 da5
gpart modify -i 3 -l disk6 da6
gpart modify -i 3 -l disk7 da7

Also remove the contents of /etc/fstab (the swap setup done by zfsinstall is not suitable for our needs:

cp /dev/null /mnt/etc/fstab

Finally run:

chroot /mnt

and do some final adjustments (IP address configuration / enable sshd etc.).

Reboot the system (don't forget to remove the mfsBSD installation medium). Note that booting from raidz2 can take a few minutes.

Swap Configuration

Even though mfsBSD creates swap partitions on all hard drives of the container, it only puts the first one into /etc/fstab. Also, swap is not encrypted (unfortunately ZFS based swap is still not recommended at this point). Since swap should be as reliable as the data storage - survive the loss of two hard drives - and also should be encrypted, the following procedure is used to create two swap partitions using gmirror and geli:

  1. Load the geom mirror module:

    kldload geom_mirror
  2. Create gmirror devices (-b prefer means, that the primary drive will always be used for reading - this is only necessary if you want store kernel crash dumps - otherwise use one of the other algorithms provided by gmirror, check the man page):

    gmirror label -b prefer -F primaryswap \
    gpt/swap0 gpt/swap4 gpt/swap2 gpt/swap6
    gmirror label -b prefer -F secondaryswap \
    gpt/swap1 gpt/swap5 gpt/swap3 gpt/swap7
  3. Put swap devices into /etc/fstab (note the .eli extension, which will cause the automatic generation of a geli encrypted swap device on boot):

    cat >>/etc/fstab <<EOF
    /dev/mirror/primaryswap.eli none swap sw 0 0
    /dev/mirror/secondaryswap.eli none swap sw 0 0
  4. Put load geom mirror module on startup into /boot/loader.conf:

    echo 'geom_mirror_load="YES"' >>/boot/loader.conf
  5. Reboot

After reboot use swapinfo to check if things worked out as expected.

Configure Periodic Scripts

The following configuration settings are added to /etc/periodic.conf to improve testing/reporting:

  1. Add ZFS status to daily report

    echo 'daily_status_zfs_enable="YES"' >>/etc/periodic.conf
  2. Add gmirror status to daily report

    echo 'daily_status_gmirror_enable="YES"' >>/etc/periodic.conf
  3. Scrub ZFS pool regularly (default is every 30 days)

    echo 'daily_scrub_zfs_enable="YES"' >>/etc/periodic.conf


Thanks to mfsBSD, setting up FreeBSD on ZFS raidz2 was a matter of minutes. In my next posts I will cover additional tasks performed while setting up the backup server, like configuring smartmontools to monitor the HDD health and some sendmail fine tuning.