Michael's Daemonic Doodles

Partition the new drive using gpart

The original setup has been done using mfsBSD, using a swap partition size of 16GB. In case one is not certain how to partition the drive, just check the setup of another drive in the pool, in this case e.g.:

[root@backup ~]# gpart list da1
Geom name: da1
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 1953525134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da1p1
   Mediasize: 65536 (64k)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17408
   Mode: r0w0e0
   rawuuid: fa3ef576-83ed-11e1-bdd5-001517783d80
   rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f
   label: (null)
   length: 65536
   offset: 17408
   type: freebsd-boot
   index: 1
   end: 161
   start: 34
2. Name: da1p2
   Mediasize: 17179869184 (16G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 82944
   Mode: r1w1e1
   rawuuid: fa45c1b1-83ed-11e1-bdd5-001517783d80
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: swap1
   length: 17179869184
   offset: 82944
   type: freebsd-swap
   index: 2
   end: 33554593
   start: 162
3. Name: da1p3
   Mediasize: 983024916992 (915G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 82944
   Mode: r1w1e1
   rawuuid: fa4f7605-83ed-11e1-bdd5-001517783d80
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: disk1
   length: 983024916992
   offset: 17179952128
   type: freebsd-zfs
   index: 3
   end: 1953525134
   start: 33554594
Consumers:
1. Name: da1
   Mediasize: 1000204886016 (931G)
   Sectorsize: 512
   Mode: r2w2e4

Using this information we can partition the new drive:

gpart create -s GPT da0
gpart add -t freebsd-boot -s 128 da0
gpart add -t freebsd-swap -s 16G -l swap0 da0
gpart add -t freebsd-zfs -l disk0 da0
dd if=/dev/zero of=/dev/da0p2 bs=512 count=560
dd if=/dev/zero of=/dev/da0p3 bs=512 count=560

(dd is to make sure old gmirror/ZFS meta data is removed).

Rebuild the gmirror swap container

It's important to make gmirror forget the missing disk, so it can be replaced. Then the new partition is inserted at position 0 (where it used to be before).

gmirror forget primaryswap
gmirror insert -p 0 primaryswap /dev/da0p2

You can check the status any time during the process:

[root@backup ~]# gmirror status
                Name    Status  Components
  mirror/primaryswap  DEGRADED  da2p2 (ACTIVE)
                                da4p2 (ACTIVE)
                                da6p2 (ACTIVE)
                                da0p2 (SYNCHRONIZING, 0%)
mirror/secondaryswap  COMPLETE  da1p2 (ACTIVE)
                                da3p2 (ACTIVE)
                                da5p2 (ACTIVE)
                                da7p2 (ACTIVE)

Once it's done the mirror is back to its normal state:

[root@backup ~]# gmirror status
                Name    Status  Components
  mirror/primaryswap  COMPLETE  da2p2 (ACTIVE)
                                da4p2 (ACTIVE)
                                da6p2 (ACTIVE)
                                da0p2 (ACTIVE)
mirror/secondaryswap  COMPLETE  da1p2 (ACTIVE)
                                da3p2 (ACTIVE)
                                da5p2 (ACTIVE)
                                da7p2 (ACTIVE)

Replace device in ZFS pool and resilver

The final step is to replace the drive in the ZFS pool (which will start resilvering automatically). Please keep in mind that resilvering is I/O intensive. In my case it took a long time and slowed down the disk subsystem's performance quite dramatically. This was primarily because of the large number of small files on the system. There are ways to throttle the resilvering process, which I might try next time - namely vfs.zfs.scrub_limit, which can be set in /boot/loader.conf.

The procedure for replacing the drive is straightforward:

zpool replace tank da0p3

You can check the status of the resilver operation anytime using zpool status:

[root@backup ~]# zpool status

  pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
     continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scan: resilver in progress since Mon May  7 20:18:34 2012
    11.7M scanned out of 908G at 353K/s, (scan is slow, no estimated time)
    1.31M resilvered, 0.00% done
config:

     NAME                        STATE     READ WRITE CKSUM
     tank                        DEGRADED     0     0     0
       raidz2-0                  DEGRADED     0     0     0
         replacing-0             REMOVED      0     0     0
           15364271088212071398  REMOVED      0     0     0  was /dev/da0p3/old
           da0p3                 ONLINE       0     0     0  (resilvering)
         da1p3                   ONLINE       0     0     0
         da2p3                   ONLINE       0     0     0
         da3p3                   ONLINE       0     0     0
         da4p3                   ONLINE       0     0     0
         da5p3                   ONLINE       0     0     0
         da6p3                   ONLINE       0     0     0
         da7p3                   ONLINE       0     0     0

errors: No known data errors

Depending on your setup this can take a substantial amount of time.

Reinstall boot code

This is easy to miss, so make sure it's done:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0