Converting a single disk ZFS pool into a ZFS mirror without rebooting
We have a couple of cheap servers using a single 1 TB drive connected to the on-board Intel SATA controller and wanted to add a second hard drive to improve reliability. The following instructions show how to convert the ZFS pool into a mirrored pool without rebooting.
Prerequisites
The machines in question run FreeBSD 8 and feature a hot-pluggable backplane. FreeBSD has been setup using mfsBSD, which used gptids in the process. The existing drive is connected to ata2, the device id is ad4. The new drive will be connected to ata3 and will appear as /dev/ad6. The pool is named tank. Swap has been encrypted using geli.
See also this blog post decribing a similar setup.
Connecting the drive
The first step is to connect the new hard drive and make the OS aware of it. This is done by detach and re-attaching the SATA channel:
[root@host ~]# atacontrol detach ata3 [root@host ~]# atacontrol attach ata3 Master: ad6 <ST1000NM0011/SN02> SATA revision 2.x Slave: no device present [root@host ~]#
Note
In FreeBSD 9, SATA devices are connected to the CAM subsystem through the ada driver. This affects device names (ada0 and ada1 instead of ad4 and ad6) as well as the command used - camcontrol rescan instead of atacontrol detach/attach.
Write the partition table
Use gpart list to figure out the existing disk layout (especially swap size). In our case the existing layout uses an 8 GB swap partition. Make sure you understand your own setup before entering any of the commands below.
Create the partition table on the new disk according to the existing:
[root@host ~]# gpart create -s GPT ad6 ad6 created [root@host ~]# gpart add -t freebsd-boot -s 128 ad6 ad6p1 added [root@host ~]# gpart add -t freebsd-swap -s 8G -l swap1 ad6 ad6p2 added [root@host ~]# gpart add -t freebsd-zfs -l disk1 ad6 ad6p3 added [root@host ~]# dd if=/dev/zero of=/dev/ad6p2 bs=512 count=560 560+0 records in 560+0 records out 286720 bytes transferred in 0.068626 secs (4178013 bytes/sec) [root@host ~]# dd if=/dev/zero of=/dev/ad6p3 bs=512 count=560 560+0 records in 560+0 records out 286720 bytes transferred in 0.029719 secs (9647740 bytes/sec) [root@host ~]#
Make sure the gpt labels of the existing drive use the same scheme (mfsBSD by default doesn't label partitions):
[root@host ~]# gpart modify -l swap0 -i 2 ad4 ad4p2 modified [root@host ~]# gpart modify -l disk0 -i 3 ad4 ad4p3 modified [root@host ~]#
Add device to zpool (creating the mirror)
Since the zpool uses gptids - probably not a bad idea in a setup where device renumbering is a constant threat - use gpart list and zpool status to figure out the correct gptids for the zpool. Since we know the disk layout, in our example the third and the sixth line are the device ids to use:
[root@host ~]# gpart list | grep rawuuid rawuuid: 81a4c14d-8bd4-11e0-8c1b-90e6ba922478 rawuuid: 81ab85f4-8bd4-11e0-8c1b-90e6ba922478 rawuuid: 81b18208-8bd4-11e0-8c1b-90e6ba922478 rawuuid: 6476c6ce-c006-11e1-bc3a-90e6ba922478 rawuuid: 6480fe17-c006-11e1-bc3a-90e6ba922478 rawuuid: 6489b265-c006-11e1-bc3a-90e6ba922478 [root@host /dev]#
Now attach the new device to the existing one to form the mirror:
[root@host ~]# zpool attach tank gptid/81b18208-8bd4-11e0-8c1b-90e6ba922478 gptid/6489b265-c006-11e1-bc3a-90e6ba922478 [root@host ~]#
Use zpool status to verify the status. Resilvering started automatically, depending on the amount of disk space used this might take several hours.
Install bootcode on new drive
Install the gptzfsbootcode on the drive, so the machine can boot from the new drive as well:
[root@host /dev]# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ad6 bootcode written to ad6 [root@host /dev]#
Change swap to use gmirror
Note
This assumes the system has enough memory left to exist without an active swap partition during the process.
Use swapinfo and geli status to determine the current swap device. Disable the swap device using swapoff, in our case:
[root@host /dev]# swapoff /dev/gptid/81ab85f4-7bd4-11e0-8c1b-90e6ba922478.eli [root@host /dev]#
Remove information of the old swap partition, so the gpt label becomes available:
[root@host /dev]# dd if=/dev/zero of=/dev/ad4p2 bs=512 count=560 560+0 records in 560+0 records out 286720 bytes transferred in 0.215379 secs (1331235 bytes/sec) [root@host /dev]# ls /dev/gpt swap0 swap1 [root@host /dev]#
Load the geom_mirror kernel module and add it to system startup:
[root@host /dev]# kldload geom_mirror [root@host /dev]# [root@host /dev]# echo 'geom_mirror_load="YES"' >>/boot/loader.conf
Configure a gmirror called "swap" using the two configured swap partitions:
[root@host /dev]# gmirror label -b prefer -F swap gpt/swap0 gpt/swap1 [root@host /dev]# ls /dev/mirror/swap /dev/mirror/swap [root@host /dev]#
Remove the old swap line in /etc/fstab and add the following line (.eli enables encryption):
/dev/mirror/swap.eli none swap sw 0 0
Attach the geli swap partition:
[root@host /dev]# service encswap start
[root@host /dev]# geli status
Name Status Components
mirror/swap.eli ACTIVE mirror/swap
[root@host /dev]#
And enable swap:
[root@host /dev]# service swap1 start [root@host /dev]# swapinfo Device 512-blocks Used Avail Capacity /dev/mirror/swap.eli 16777208 0 16777208 0% [root@host /dev]#
Further tasks
In case smartmontools are used (which is something we do) the new device should be added to /etc/periodic.conf, e.g.
daily_status_smart_devices="ad4 ad6"
If you're running smartd, /usr/local/etc/rc.d/smartd.conf should be configured to include the new drive.
Conclusion
Turning a single disk zpool into a mirror is pretty simple and doesn't require stopping any services or rebooting the server. Be aware though, that resilvering will have a temporary impact on disk throughput, so avoid performing this procedure during peek hours.
