Michael's Daemonic Doodles

...blogging bits of BSD

Converting a single disk ZFS pool into a ZFS mirror without rebooting

We have a couple of cheap servers using a single 1 TB drive connected to the on-board Intel SATA controller and wanted to add a second hard drive to improve reliability. The following instructions show how to convert the ZFS pool into a mirrored pool without rebooting.


The machines in question run FreeBSD 8 and feature a hot-pluggable backplane. FreeBSD has been setup using mfsBSD, which used gptids in the process. The existing drive is connected to ata2, the device id is ad4. The new drive will be connected to ata3 and will appear as /dev/ad6. The pool is named tank. Swap has been encrypted using geli.

See also this blog post decribing a similar setup.

Connecting the drive

The first step is to connect the new hard drive and make the OS aware of it. This is done by detach and re-attaching the SATA channel:

[root@host ~]# atacontrol detach ata3
[root@host ~]# atacontrol attach ata3
Master:  ad6 <ST1000NM0011/SN02> SATA revision 2.x
Slave:       no device present
[root@host ~]#


In FreeBSD 9, SATA devices are connected to the CAM subsystem through the ada driver. This affects device names (ada0 and ada1 instead of ad4 and ad6) as well as the command used - camcontrol rescan instead of atacontrol detach/attach.

Write the partition table

Use gpart list to figure out the existing disk layout (especially swap size). In our case the existing layout uses an 8 GB swap partition. Make sure you understand your own setup before entering any of the commands below.

Create the partition table on the new disk according to the existing:

[root@host ~]# gpart create -s GPT ad6
ad6 created
[root@host ~]# gpart add -t freebsd-boot -s 128 ad6
ad6p1 added
[root@host ~]# gpart add -t freebsd-swap -s 8G -l swap1 ad6
ad6p2 added
[root@host ~]# gpart add -t freebsd-zfs -l disk1 ad6
ad6p3 added
[root@host ~]# dd if=/dev/zero of=/dev/ad6p2 bs=512 count=560
560+0 records in
560+0 records out
286720 bytes transferred in 0.068626 secs (4178013 bytes/sec)
[root@host ~]# dd if=/dev/zero of=/dev/ad6p3 bs=512 count=560
560+0 records in
560+0 records out
286720 bytes transferred in 0.029719 secs (9647740 bytes/sec)
[root@host ~]#

Make sure the gpt labels of the existing drive use the same scheme (mfsBSD by default doesn't label partitions):

[root@host ~]# gpart modify -l swap0 -i 2 ad4
ad4p2 modified
[root@host ~]# gpart modify -l disk0 -i 3 ad4
ad4p3 modified
[root@host ~]#

Add device to zpool (creating the mirror)

Since the zpool uses gptids - probably not a bad idea in a setup where device renumbering is a constant threat - use gpart list and zpool status to figure out the correct gptids for the zpool. Since we know the disk layout, in our example the third and the sixth line are the device ids to use:

[root@host ~]# gpart list | grep rawuuid
   rawuuid: 81a4c14d-8bd4-11e0-8c1b-90e6ba922478
   rawuuid: 81ab85f4-8bd4-11e0-8c1b-90e6ba922478
   rawuuid: 81b18208-8bd4-11e0-8c1b-90e6ba922478
   rawuuid: 6476c6ce-c006-11e1-bc3a-90e6ba922478
   rawuuid: 6480fe17-c006-11e1-bc3a-90e6ba922478
   rawuuid: 6489b265-c006-11e1-bc3a-90e6ba922478
[root@host /dev]#

Now attach the new device to the existing one to form the mirror:

[root@host ~]# zpool attach tank gptid/81b18208-8bd4-11e0-8c1b-90e6ba922478
[root@host ~]#

Use zpool status to verify the status. Resilvering started automatically, depending on the amount of disk space used this might take several hours.

Install bootcode on new drive

Install the gptzfsbootcode on the drive, so the machine can boot from the new drive as well:

[root@host /dev]# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ad6
bootcode written to ad6
[root@host /dev]#

Change swap to use gmirror


This assumes the system has enough memory left to exist without an active swap partition during the process.

Use swapinfo and geli status to determine the current swap device. Disable the swap device using swapoff, in our case:

[root@host /dev]# swapoff /dev/gptid/81ab85f4-7bd4-11e0-8c1b-90e6ba922478.eli
[root@host /dev]#

Remove information of the old swap partition, so the gpt label becomes available:

[root@host /dev]# dd if=/dev/zero of=/dev/ad4p2 bs=512 count=560
560+0 records in
560+0 records out
286720 bytes transferred in 0.215379 secs (1331235 bytes/sec)

[root@host /dev]# ls /dev/gpt
swap0 swap1
[root@host /dev]#

Load the geom_mirror kernel module and add it to system startup:

[root@host /dev]# kldload geom_mirror
[root@host /dev]#

[root@host /dev]# echo 'geom_mirror_load="YES"' >>/boot/loader.conf

Configure a gmirror called "swap" using the two configured swap partitions:

[root@host /dev]# gmirror label -b prefer -F swap gpt/swap0 gpt/swap1
[root@host /dev]# ls /dev/mirror/swap
[root@host /dev]#

Remove the old swap line in /etc/fstab and add the following line (.eli enables encryption):

/dev/mirror/swap.eli none swap sw 0 0

Attach the geli swap partition:

[root@host /dev]# service encswap start
[root@host /dev]# geli status
           Name  Status  Components
mirror/swap.eli  ACTIVE  mirror/swap
[root@host /dev]#

And enable swap:

[root@host /dev]# service swap1 start
[root@host /dev]# swapinfo
Device          512-blocks     Used    Avail Capacity
/dev/mirror/swap.eli   16777208        0 16777208     0%
[root@host /dev]#

Further tasks

In case smartmontools are used (which is something we do) the new device should be added to /etc/periodic.conf, e.g.

daily_status_smart_devices="ad4 ad6"

If you're running smartd, /usr/local/etc/rc.d/smartd.conf should be configured to include the new drive.


Turning a single disk zpool into a mirror is pretty simple and doesn't require stopping any services or rebooting the server. Be aware though, that resilvering will have a temporary impact on disk throughput, so avoid performing this procedure during peek hours.