Michael's Daemonic Doodles

...blogging bits of BSD

Shrinking a ZFS root pool (HDD to SSD migration)

I recently started migrating servers with relatively low storage space requirements to SSDs. In many cases the HDDs that get replaced are much bigger than required and unfortunately the zpools have been configured to use all the available space. Since shrinking a pool is not supported by ZFS directly, the procedure is a little more elaborate.

Preconditions & caveats

This assumes that your running a ZFS root pool named oldpool using a "classic" disk layout, e.g.:

# gpart show
=>       34  488397101  ada0  GPT  (232G)
         34        128     1  freebsd-boot  (64k)
        162    8388608     2  freebsd-swap  (4.0G)
    8388770  480008365     3  freebsd-zfs  (228G)

And a simple pool layout, e.g.:

NAME               USED  AVAIL  REFER  MOUNTPOINT
oldpool            659M   224G    21K  none
oldpool/root       659M   224G   421M  /
oldpool/root/tmp  2.15M   224G  2.15M  /tmp
oldpool/root/usr   182M   224G   182M  /usr
oldpool/root/var  54.3M   224G  54.3M  /var

It also assumes that you're replacing one disk with another (HDD to SSD in this case).

This is a single disk pool, the general principle works regardless of the type of pool used (mirror, raidz...). /dev/ada0 is the existing drive, /dev/ada1 the new SSD drive. This ignores trying to force 4K block sizes (ashift 12).

Creating a new, smaller pool

Create partitions and install ZFS boot code

gpart create -s GPT ada1
gpart add -t freebsd-boot -s 128 ada1
gpart add -t freebsd-swap -s 4G -l newswap ada1
gpart add -t freebsd-zfs -s 105G -l newdisk ada1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

Create new zpool

zpool create -o cachefile=/tmp/zpool.cache newpool gpt/newdisk

Take a snapshot and transfer it new the new pool

zfs snapshot -r oldpool@shrink
zfs send -vR oldpool@shrink | zfs receive -vFd newpool
zfs destroy -r oldpool@shrink
zfs destroy -r newpool@shrink

Make new zpool bootable

zpool set bootfs=newpool/root newpool

Export and re-import while preserving cache

This mounts the new pool at /mnt:

cp /tmp/zpool.cache /tmp/newpool.cache
zpool export newpool
zpool import -c /tmp/newpool.cache -R /mnt newpool
zfs set mountpoint=/ newpool/root
cp /tmp/newpool.cache /mnt/boot/zfs/zpool.cache

Getting rid of legacy mounts

In case your old pool still used legacy mounts this is your chance to get rid of them. This is done by:

  1. Removing all mount entries for the pool from /mnt/etc/fstab

  2. Making sure that zfs_enable="YES" is part of /mnt/etc/rc.conf

  3. Making sure that mountpoints are inherited, e.g.:

    zfs inherit mountpoint newpool/root/tmp
    zfs inherit mountpoint newpool/root/var
    zfs inherit mountpoint newpool/root/usra
    

Configure swap

/mnt/etc/fstab should contain:

/dev/gpt/newswap.eli none swap sw 0 0

This will create an encrypted swap partition. If you prefer unencrypted swap simply remove .eli from the device name.

Modify loader configuration

Only do this if you want to keep the new pool name.

Change vfs.root.mountfrom in /mnt/boot/loader.conf so it points to the correct root partition:

zfs_load="YES"
vfs.root.mountfrom="zfs:newpool/root"

Done?

If you decided to keep the new pool name, you're basically done now. Shutdown the computer, remove the old drive and boot up again.

Renaming the pool to its original name

To rename newpool to oldpool, boot from removable media. I prefer mfsBSD for this purpose, but any live FreeBSD image will do.

After booting the image do:

zpool import -o cachefile=/tmp/zpool.cache -R /mnt newpool oldpool
cp /tmp/zpool.cache /mnt/boot/zfs/.
zpool set bootfs=tank/root tank
reboot

Conclusion

Shrinking a ZFS pool is quite some effort. Save yourself the hassle by not oversizing your pools. You can always expand them later easily using gpart modify and zpool online -e pool device.