As I explained in an earlier post, we had issues with one of our backup servers. I decided to do a complete reinstall and ended up using an LSI SAS 9211 controller running in initiator-target mode (IT firmware), so all eight disks show up as plain HBAs and can be used to form a zpool. After considering my options I decided to go with raidz2, which will provide a good balance of storage space and safety.
Since I had great results using Martin Matuška's brilliant mfsBSD collection in the past, I also used it for setting up FreeBSD 9.0 on the backup host. Custom made versions of mfsBSD are really easy to create and we used a specially hardened custom distribution of mfsBSD for a PCI DSS compliant setup in the past. In this case the precompiled 9.0-RELEASE amd64 special edition, which can downloaded from the project's home page, works just fine.
After booting mfsBSD 9.0-RELEASE amd64 special edition, login using the credentials root/mfsroot. Next, mount the CD containing installation data:
mount -t cd9660 /dev/cd0 /cdrom
The exact name of the device depends on the installation medium used.
Unfortunately at this point zfsinstall doesn't support raidz out of the box. This is easily fixed by modifying the script:
sed -i .old s/raidz/raidz2/g /root/bin/zfsinstall
Now zfsinstall can be run (this assumes that there is no existing ZFS container on the drives, drive names are /dev/da[0-8], 16GB of swap).
zfsinstall -t /cdrom/9.0-RELEASE-amd64.tar.xz -s 16G -r raidz2 \ -d da0 -d da1 -d da2 -d da3 -d da4 -d da5 -d da6 -d da7
Zfsinstall doesn't label the partitions, to make life easier it's best to manually label swap partitions ...
gpart modify -i 2 -l swap0 da0 gpart modify -i 2 -l swap1 da1 gpart modify -i 2 -l swap2 da2 gpart modify -i 2 -l swap3 da3 gpart modify -i 2 -l swap4 da4 gpart modify -i 2 -l swap5 da5 gpart modify -i 2 -l swap6 da6 gpart modify -i 2 -l swap7 da7
... and ZFS pool members:
gpart modify -i 3 -l disk0 da0 gpart modify -i 3 -l disk1 da1 gpart modify -i 3 -l disk2 da2 gpart modify -i 3 -l disk3 da3 gpart modify -i 3 -l disk4 da4 gpart modify -i 3 -l disk5 da5 gpart modify -i 3 -l disk6 da6 gpart modify -i 3 -l disk7 da7
Also remove the contents of /etc/fstab (the swap setup done by zfsinstall is not suitable for our needs:
cp /dev/null /mnt/etc/fstab
and do some final adjustments (IP address configuration / enable sshd etc.).
Reboot the system (don't forget to remove the mfsBSD installation medium). Note that booting from raidz2 can take a few minutes.
Even though mfsBSD creates swap partitions on all hard drives of the container, it only puts the first one into /etc/fstab. Also, swap is not encrypted (unfortunately ZFS based swap is still not recommended at this point). Since swap should be as reliable as the data storage - survive the loss of two hard drives - and also should be encrypted, the following procedure is used to create two swap partitions using gmirror and geli:
Load the geom mirror module:kldload geom_mirror
Create gmirror devices (-b prefer means, that the primary drive will always be used for reading - this is only necessary if you want store kernel crash dumps - otherwise use one of the other algorithms provided by gmirror, check the man page):gmirror label -b prefer -F primaryswap \ gpt/swap0 gpt/swap4 gpt/swap2 gpt/swap6 gmirror label -b prefer -F secondaryswap \ gpt/swap1 gpt/swap5 gpt/swap3 gpt/swap7
Put swap devices into /etc/fstab (note the .eli extension, which will cause the automatic generation of a geli encrypted swap device on boot):cat >>/etc/fstab <<EOF /dev/mirror/primaryswap.eli none swap sw 0 0 /dev/mirror/secondaryswap.eli none swap sw 0 0 EOF
Put load geom mirror module on startup into /boot/loader.conf:echo 'geom_mirror_load="YES"' >>/boot/loader.conf
After reboot use swapinfo to check if things worked out as expected.
Configure Periodic Scripts
The following configuration settings are added to /etc/periodic.conf to improve testing/reporting:
Add ZFS status to daily reportecho 'daily_status_zfs_enable="YES"' >>/etc/periodic.conf
Add gmirror status to daily reportecho 'daily_status_gmirror_enable="YES"' >>/etc/periodic.conf
Scrub ZFS pool regularly (default is every 30 days)echo 'daily_scrub_zfs_enable="YES"' >>/etc/periodic.conf
Thanks to mfsBSD, setting up FreeBSD on ZFS raidz2 was a matter of minutes. In my next posts I will cover additional tasks performed while setting up the backup server, like configuring smartmontools to monitor the HDD health and some sendmail fine tuning.