Running FreeBSD on a Raspberry Pi3 using a custom image created with crochet and poudriere
I used to download images from raspbsd.org and install binary packages using pkg on the device. This has some disadvantages:
- The image there is updated infrequently
- Installing using pkg is relatively slow
- Binary packages aren't always in sync when running CURRENT
- Building things yourself gives you more control
Using crochet and poudriere it's quite easy to create an image with pre-installed packages, which helps a lot when provisioning a large number of devices. Packages are cross-compiled using qemu, which is slow, but still much faster than building on the device itself.
Preconditions
The procedure below was done on a system running FreeBSD 12.0-BETA1 r339435 GENERIC amd64 using binary packages through pkg. The resulting image is 6GB (6 * 10^9 bytes) in size and will expand on first boot using growfs. It uses DHCP on its ethernet port, remote ssh credentials are user raspberry, password raspberry.
The procedure was tested using sh and bash, some constructs (like "EOF") work differently in other shells.
Install basic packages
pkg install git poudriere qemu-user-static rpi-firmware subversion u-boot-rpi3
Checkout crochet
git clone https://github.com/freebsd/crochet cd crochet
Create list of packages to add to the image
Only a few basic packages to serve as an example:
cat >pkglist <<EOF editors/joe ftp/curl mail/opensmtpd security/sudo shells/bash sysutils/tmux EOF
Configure poudriere
Assumes a zpool named zroot and enables ccache:
sysrc -f /usr/local/etc/poudriere.conf ZPOOL=zroot sysrc -f /usr/local/etc/poudriere.conf CCACHE_DIR=/var/cache/ccache mkdir -p /var/cache/ccache mkdir -p /usr/ports/distfiles
Configure binary image activator
This is key to automatically invoke qemu-aarch64-static to run ARM binaries:
binmiscctl add aarch64 \ --interpreter "/usr/local/bin/qemu-aarch64-static" \ --magic "\x7f\x45\x4c\x46\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00" \ --mask "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff" \ --size 20 --set-enabled
Important
This loads the imgact_binmisc kernel module. You need to do this again after each reboot. The remainder of the procedure depends on this.
Create poudriere jail and ports tree
poudriere jail -c -j 12aarch64 -a arm64.aarch64 -v 12.0-BETA1 poudriere ports -c -p rpi3
Fine tune package build make.conf
For this build we're using security/libressl instead of base's openssl (mail/opensmtpd wouldn't build with system openssl at the moment):
cat >/usr/local/etc/poudriere.d/12aarch64-make.conf <<"EOF"
DEFAULT_VERSIONS+= ssl=libressl
.if ${.CURDIR:M*/ftp/curl}
OPTIONS_FILE_UNSET+=TLS_SRP
.endif
EOF
Build packages
This takes a long time:
poudriere bulk -j 12aarch64 -p rpi3 -f pkglist
Configure crochet
Important
Having qemu-user-static installed and adding it through the configuration below are required to end up with correct packages installed. You also need to make sure the binary image activator is configured correctly like described above, otherwise POST_INSTALL scripts will fail (there will be error messages, but the image build will finish anyway and you'll basically end up with a corrupted package installation).
cat >config.rpi3.sh <<"EOF"
board_setup RaspberryPi3
option ImageSize 6gb
option Growfs
option User raspberry
FREEBSD_SRC=${TOPDIR}/src
option PackageInit http://localhost:8080
option Package $(cat pkglist)
add_qemu ( ) {
echo "Installing qemu-aarch64-static"
mkdir -p ${BOARD_FREEBSD_MOUNTPOINT}/usr/local/bin
cp -av /usr/local/bin/qemu-aarch64-static ${BOARD_FREEBSD_MOUNTPOINT}/usr/local/bin/.
}
remove_qemu ( ) {
echo "Removing qemu-aarch64-static"
rm -v ${BOARD_FREEBSD_MOUNTPOINT}/usr/local/bin/qemu-aarch64-static
}
PRIORITY=50 strategy_add $PHASE_FREEBSD_OPTION_INSTALL add_qemu
PRIORITY=150 strategy_add $PHASE_FREEBSD_OPTION_INSTALL remove_qemu
EOF
Check out FreeBSD source tree
svn checkout https://svn.freebsd.org/base/stable/12 src
Activate python web server to serve local packages
This will serve packages created by poudriere over http (binding to 0.0.0.0:8080):
cd /usr/local/poudriere/data/packages/12aarch64-rpi3 python2.7 -m SimpleHTTPServer 8080 & cd -
Note
Don't forget to stop it again once you're done. You can also create a more proper setup using ssh or https (e.g. nginx) and use that to serve package updates to your devices.
Build the image
This will take a while:
./crochet.sh -c config.rpi3.sh
The resulting image can be written to an SD card using dd:
dd if=work/FreeBSD-aarch64-12-GENERIC-RaspberryPi3.img bs=1m of=/dev/daX
(replace daX with the actual device name of your SD card).
