Michael's Daemonic Doodles

mfsBSD

Since I had great results using Martin Matuška's brilliant mfsBSD collection in the past, I also used it for setting up FreeBSD 9.0 on the backup host. Custom made versions of mfsBSD are really easy to create and we used a specially hardened custom distribution of mfsBSD for a PCI DSS compliant setup in the past. In this case the precompiled 9.0-RELEASE amd64 special edition, which can downloaded from the project's home page, works just fine.

Basic Installation

After booting mfsBSD 9.0-RELEASE amd64 special edition, login using the credentials root/mfsroot. Next, mount the CD containing installation data:

mount -t cd9660 /dev/cd0 /cdrom

Unfortunately at this point zfsinstall doesn't support raidz out of the box. This is easily fixed by modifying the script:

sed -i .old s/raidz/raidz2/g /root/bin/zfsinstall

Now zfsinstall can be run (this assumes that there is no existing ZFS container on the drives, drive names are /dev/da[0-8], 16GB of swap).

zfsinstall -t /cdrom/9.0-RELEASE-amd64.tar.xz -s 16G -r raidz2 \
-d da0 -d da1 -d da2 -d da3 -d da4 -d da5 -d da6 -d da7

Zfsinstall doesn't label the partitions, to make life easier it's best to manually label swap partitions ...

gpart modify -i 2 -l swap0 da0
gpart modify -i 2 -l swap1 da1
gpart modify -i 2 -l swap2 da2
gpart modify -i 2 -l swap3 da3
gpart modify -i 2 -l swap4 da4
gpart modify -i 2 -l swap5 da5
gpart modify -i 2 -l swap6 da6
gpart modify -i 2 -l swap7 da7

... and ZFS pool members:

gpart modify -i 3 -l disk0 da0
gpart modify -i 3 -l disk1 da1
gpart modify -i 3 -l disk2 da2
gpart modify -i 3 -l disk3 da3
gpart modify -i 3 -l disk4 da4
gpart modify -i 3 -l disk5 da5
gpart modify -i 3 -l disk6 da6
gpart modify -i 3 -l disk7 da7

Also remove the contents of /etc/fstab (the swap setup done by zfsinstall is not suitable for our needs:

cp /dev/null /mnt/etc/fstab

Finally run:

chroot /mnt

and do some final adjustments (IP address configuration / enable sshd etc.).

Reboot the system (don't forget to remove the mfsBSD installation medium). Note that booting from raidz2 can take a few minutes.

Swap Configuration

Even though mfsBSD creates swap partitions on all hard drives of the container, it only puts the first one into /etc/fstab. Also, swap is not encrypted (unfortunately ZFS based swap is still not recommended at this point). Since swap should be as reliable as the data storage - survive the loss of two hard drives - and also should be encrypted, the following procedure is used to create two swap partitions using gmirror and geli:

1. Load the geom mirror module:

   kldload geom_mirror
   

2. Create gmirror devices (-b prefer means, that the primary drive will always be used for reading - this is only necessary if you want store kernel crash dumps - otherwise use one of the other algorithms provided by gmirror, check the man page):

   gmirror label -b prefer -F primaryswap \
   gpt/swap0 gpt/swap4 gpt/swap2 gpt/swap6
   
   gmirror label -b prefer -F secondaryswap \
   gpt/swap1 gpt/swap5 gpt/swap3 gpt/swap7
   

3. Put swap devices into /etc/fstab (note the .eli extension, which will cause the automatic generation of a geli encrypted swap device on boot):

   cat >>/etc/fstab <<EOF
   /dev/mirror/primaryswap.eli none swap sw 0 0
   /dev/mirror/secondaryswap.eli none swap sw 0 0
   EOF
   

4. Put load geom mirror module on startup into /boot/loader.conf:

   echo 'geom_mirror_load="YES"' >>/boot/loader.conf
   

5. Reboot

After reboot use swapinfo to check if things worked out as expected.

Configure Periodic Scripts

The following configuration settings are added to /etc/periodic.conf to improve testing/reporting:

1. Add ZFS status to daily report

   echo 'daily_status_zfs_enable="YES"' >>/etc/periodic.conf
   

2. Add gmirror status to daily report

   echo 'daily_status_gmirror_enable="YES"' >>/etc/periodic.conf
   

3. Scrub ZFS pool regularly (default is every 30 days)

   echo 'daily_scrub_zfs_enable="YES"' >>/etc/periodic.conf
   

Conclusion

Thanks to mfsBSD, setting up FreeBSD on ZFS raidz2 was a matter of minutes. In my next posts I will cover additional tasks performed while setting up the backup server, like configuring smartmontools to monitor the HDD health and some sendmail fine tuning.