Immutable root with atomic upgrades
What?
This article provides a basic guide to setting up a read-only-root-based Alpine Linux system with several boot environments and atomic upgrades using rEFInd and btrfs.
Why?
Read-only root and atomic upgrades with ability to easily rollback or boot previous configurations is a concept that got some popularity recently. Distributions providing and promoting such features, for example, are Fedora Silverblue, Opensuse MicroOS, NixOS and GNU Guix.
While Alpine Linux has it's killer features it lacks mentioned above ones on default setup. This is a proof of concept that it's possible to implement them in a minimal way on a minimal system.
Preparation
You should have bootable Alpine media. The process to obtain it decribed on the installation page.
Partitioning disks
In this guide it is assumed that you have a fresh UEFI system without OS and just managed to boot into live Alpine using USB flash drive or CD.
The first step is creating partition table on your HDD/SSD target device (/dev/sda here):
# apk add gptfdisk # gdisk /dev/sda > o ↵ > y ↵ > w ↵ > y ↵
Now we can define the partitions:
# cgdisk /dev/sda
Partition creation process consists of several steps:
- Start sector - you can safely use default value by pressing ↵
- Size
- Type (as hex code) - EFI is ef00, Linux filesystem is 8300, Swap is 8200.
Result table:
Part. # Size Partition Type Partition Name ---------------------------------------------------------------- 1 200.0 MiB EFI System EFI 2 200.0 GiB Linux filesystem ROOT 3 32.0 GiB Linux swap SWAP
ROOT partition name will later be used in rEFInd configuration to identify boot volume.
Next step is creating filesystems:
# mkfs.vfat -F32 /dev/sda1 # mkfs.btrfs /dev/sda2 # mkswap /dev/sda3
Now we can mount our root volume:
# mount -t btrfs /dev/sda2 /mnt
File system structure
Now we should create file structure that would provide reliable atomic system upgrades.
Start with following directories:
# mkdir /mnt/next
Stores next current link, is necessary due to how busybox mv does atomic link replacement.
# mkdir /mnt/commons
Stores common non-snapshotting subvolumes.
We may populate it right away:
# btrfs subvolume create /mnt/commons/@var@tmp # btrfs subvolume create /mnt/commons/@var@cache # btrfs subvolume create /mnt/commons/@var@log # btrfs subvolume create /mnt/commons/@home
If you use flatpak, you may also want to keep it's directory separate:
# btrfs subvolume create /mnt/commons/@var@lib@flatpak
Include anything else in /var that should be mutable, for example:
# btrfs subvolume create /mnt/commons/@var@lib@iwd
Next, most important directories:
# mkdir /mnt/snapshots
Stores directories containing snapshots belonging to one generation.
# mkdir /mnt/links
Stores generations of directories containing links to snapshot generations.
Let's create first generation and populate it with one OS root snapshot @:
# NEWSNAPSHOTS="$(date -u +"%Y%m%d%H%M%S")$(cat /dev/urandom | tr -dc 'a-zA-Z' | fold -w 8 | head -n 1)" # mkdir "/mnt/snapshots/$NEWSNAPSHOTS" # btrfs subvolume create /mnt/snapshots/$NEWSNAPSHOTS/@
Populate links:
# NEWLINKS="$(date -u +"%Y%m%d%H%M%S")$(cat /dev/urandom | tr -dc 'a-zA-Z' | fold -w 8 | head -n 1)" # mkdir "/mnt/links/$NEWLINKS" # ln -s "../../snapshots/$NEWSNAPSHOTS" "/mnt/links/$NEWLINKS/0" # ln -s "../../snapshots/$NEWSNAPSHOTS" "/mnt/links/$NEWLINKS/1" # ln -s "../../snapshots/$NEWSNAPSHOTS" "/mnt/links/$NEWLINKS/2" # ln -s "../../snapshots/$NEWSNAPSHOTS" "/mnt/links/$NEWLINKS/3"
You can have as many links as you like, just apply changes to rEFInd config and upgrade scripts described below accordingly.
Link that will point to latest links generation:
# ln -s "./links/$NEWLINKS" /mnt/current
This setup allows us to just have static rEFInd config that points to to /current/0/@, /current/1/@, etc. while the actual underlying boot environment will change with each upgrade.
But how will fs mounting services know which snapshot generation is currently loaded?
The answer is common fstab in the btrfs root.
Get UUIDs of the partitions first:
# blkid > /mnt/fstab
Now edit fstab accordingly:
# vi /mnt/fstab
Example:
UUID=b9ff5e7b-e128-4e64-861a-2fdd794a9828 / btrfs subvol=CURRENT_SNAPSHOTS_PATH/@,ro,noatime 0 0 UUID=b9ff5e7b-e128-4e64-861a-2fdd794a9828 /var/tmp btrfs subvol=/commons/@var@tmp,rw,noatime 0 0 UUID=b9ff5e7b-e128-4e64-861a-2fdd794a9828 /var/cache btrfs subvol=/commons/@var@cache,rw,noatime 0 0 UUID=b9ff5e7b-e128-4e64-861a-2fdd794a9828 /var/log btrfs subvol=/commons/@var@log,rw,noatime 0 0 UUID=b9ff5e7b-e128-4e64-861a-2fdd794a9828 /var/lib/flatpak btrfs subvol=/commons/@var@lib@flatpak,rw,noatime 0 0 UUID=b9ff5e7b-e128-4e64-861a-2fdd794a9828 /var/lib/iwd btrfs subvol=/commons/@var@lib@iwd,rw,noatime 0 0 UUID=b9ff5e7b-e128-4e64-861a-2fdd794a9828 /home btrfs subvol=/commons/@home,rw,noatime 0 0 # UUID=2FE6-837A /boot/efi vfat rw,noatime,discard 0 2 tmpfs /tmp tmpfs mode=1777,noatime,nosuid,nodev,size=2G 0 0 UUID=f0239163-9d46-47c1-67a4-3ee1d63d0676 swap swap rw,noatime,discard 0 0
CURRENT_SNAPSHOTS_PATH will be replaced by scripts with, for example, /snapshots/20210411212549sdBXyLxg, and the result will be piped into /etc/fstab of a created @ snapshot during new generation preparations.
Root btrfs volume structure mounted on /mnt:
|--mnt | |--commons | | |--@var@tmp | | |--@var@cache | | |--@var@log | | |--@var@lib@flatpak | | |--@var@lib@iwd | | |--@home | |--current | |--fstab | |--links | | |--20210411213742qwrXAJBz | | | |--0 | | | |--1 | | | |--2 | | | |--3 | |--next | |--snapshots | | |--20210411212549sdBXyLxg | | | |--@
Base system install
With the directory strtucture prepared we can start installation of a basic Alpine Linux system.
Considering that installation is done from Alpine system, we only need following parts of the process:
# apk -X https://dl-cdn.alpinelinux.org/alpine/latest-stable/main -U --allow-untrusted -p /mnt/snapshots/20210411212549sdBXyLxg/@ --initdb add alpine-base
Now we can setup basic chroot to complete the installation process:
# export SNP="/mnt/snapshots/20210411212549sdBXyLxg/@" # mount -o bind /dev $SNP/dev # mount -t proc none $SNP/proc # mount -t sysfs sys $SNP/sys # sed "s#CURRENT_SNAPSHOTS_PATH#/snapshots/20210411212549sdBXyLxg#g" /mnt/fstab > "$SNP/etc/fstab" # cp -L /etc/resolv.conf "$SNP/etc/" # chroot "$SNP" /bin/sh # mount -a # mv /etc/resolv.conf /tmp/ # ln -s /tmp/resolv.conf /etc/resolv.conf
As soon as you in chroot, define repositories:
# echo "https://dl-cdn.alpinelinux.org/alpine/latest-stable/main" > /etc/apk/repositories
Example shows only main, but you should also add testing and community if you need any packages in those.
Now it's time for firmware, kernel and btrfs packages:
# apk add -U linux-firmware linux-lts btrfs-progs
You may want to change linux-firmware to a custom set of firmware packages suitable for you system, for example linux-firmware-amd linux-firmware-amd-ucode linux-firmware-amdgpu linux-firmware-ath10k linux-firmware-qca for typical AMD laptop.
It is also important to add btrfs feature to mkinitfs.conf and run mkinitfs manually:
# vi /etc/mkinitfs/mkinitfs.conf # mkinitfs
These steps prepare kernel and generate initramfs which later will be used to boot from our first snapshot.
After that you should install any package you may need on first boot.
iwd in this example, so you will not end up severed from network on your first boot.
openresolv to support changing network connection. In this case /etc/resolvconf.conf should have resolv_conf=/tmp/resolv.conf, /etc/resolv.conf should be moved to /tmp/resolv.conf and a link to the new resolv.conf location should be created: ln -sfn /tmp/resolv.conf /etc/resolv.conf.
You may also use static DNS, but this would make your network activity directly identifiable to the DNS server provider, therefore it's not recommended.
Now, configure the system, start with setting a password for the root:
# passwd root
With the snapshot prepared and configured we can chroot out of it and unmount everything:
# umount -a # exit
Finish editing snapshot by setting ro flag and unmounting root volume:
# btrfs property set -ts "/mnt/snapshots/20210411212549sdBXyLxg/@" ro true # umount /mnt
Bootloader installation
Mount the EFI partition:
# mount -t vfat /dev/sda1 /mnt # mkdir /mnt/EFI
Unpack prepared rEFInd archive and copy relevant files to /mnt/EFI/
# unzip refind-bin-version.zip # cp -r refind-bin-version/refind /mnt/EFI/ # cd /mnt/EFI/refind
Delete all unnecessary files - everything that is not for your CPU architecture:
# rm -r drivers_aa64 drivers_ia32 tools_aa64 tools_ia32 refind_aa64.efi refind_ia32.efi
Rename config file and edit it:
# mv refind.conf-sample refind.conf # vi refind.conf
And append following to the end of the file, remember to replace example UUIDs with your own for root (btrfs partition) and resume (swap partition):
menuentry "Alpine Linux" {
icon /EFI/refind/icons/os_linux.png
volume "ROOT"
loader /current/0/@/boot/vmlinuz-lts
initrd /current/0/@/boot/initramfs-lts
options "root=UUID=b9ff5e7b-e128-4e64-861a-2fdd794a9828 rootfstype=btrfs rootflags=subvol=/current/0/@,ro,noatime resume=UUID=f0239163-9d46-47c1-67a4-3ee1d63d0676 quiet splash"
submenuentry "Boot fallback 1" {
loader /current/1/@/boot/vmlinuz-lts
initrd /current/1/@/boot/initramfs-lts
options "root=UUID=b9ff5e7b-e128-4e64-861a-2fdd794a9828 rootfstype=btrfs rootflags=subvol=/current/1/@,ro,noatime resume=UUID=f0239163-9d46-47c1-67a4-3ee1d63d0676 quiet splash"
}
submenuentry "Boot fallback 2" {
loader /current/2/@/boot/vmlinuz-lts
initrd /current/2/@/boot/initramfs-lts
options "root=UUID=b9ff5e7b-e128-4e64-861a-2fdd794a9828 rootfstype=btrfs rootflags=subvol=/current/2/@,ro,noatime resume=UUID=f0239163-9d46-47c1-67a4-3ee1d63d0676 quiet splash"
}
submenuentry "Boot fallback 3" {
loader /current/3/@/boot/vmlinuz-lts
initrd /current/3/@/boot/initramfs-lts
options "root=UUID=b9ff5e7b-e128-4e64-861a-2fdd794a9828 rootfstype=btrfs rootflags=subvol=/current/3/@,ro,noatime resume=UUID=f0239163-9d46-47c1-67a4-3ee1d63d0676 quiet splash"
}
}
"ROOT" is the PARTLABEL of the btrfs partition. You may also use PARTUUID instead. To get both blkid from blkid package can be used. blkid included in busybox does not provide this information. To add rEFInd to UEFI, efibootmgr is a suitable tool:
# apk add efibootmgr # efibootmgr --create --disk /dev/sda --part 1 --loader /EFI/refind/refind_x64.efi --label "rEFInd" --verbose
/dev/sda is our disk device and 1 is the number of partition containing rEFInd.
Updating or altering the system
# touch /sbin/sysmut # chmod +x /sbin/sysmut # vi /sbin/sysmut
Example script to mutate the the system:
#!/bin/sh ROOTVOLMNT="/vol/root" && \ mkdir -p "$ROOTVOLMNT" && \ mount -t btrfs -o rw,noatime UUID=b9ff5e7b-e128-4e64-861a-2fdd794a9828 "$ROOTVOLMNT" && \ NEWSNAPSHOTS="$(date -u +"%Y%m%d%H%M%S")$(cat /dev/urandom | tr -dc 'a-zA-Z' | fold -w 8 | head -n 1)" && \ mkdir -p "$ROOTVOLMNT/snapshots/$NEWSNAPSHOTS" && \ btrfs subvolume snapshot "$ROOTVOLMNT/current/0/@" "$ROOTVOLMNT/snapshots/$NEWSNAPSHOTS/@" && \ sed "s#CURRENT_SNAPSHOTS_PATH#/snapshots/$NEWSNAPSHOTS#g" "$ROOTVOLMNT/fstab" > "$ROOTVOLMNT/snapshots/$NEWSNAPSHOTS/@/etc/fstab" && \ mount -t proc none "$ROOTVOLMNT/snapshots/$NEWSNAPSHOTS/@/proc" && \ mount -t sysfs sys "$ROOTVOLMNT/snapshots/$NEWSNAPSHOTS/@/sys" && \ mount -o bind /dev "$ROOTVOLMNT/snapshots/$NEWSNAPSHOTS/@/dev" && \ chroot "$ROOTVOLMNT/snapshots/$NEWSNAPSHOTS/@" /bin/sh -c 'mount -a; sh; APPLY=$?; umount -a; exit $APPLY' && \ btrfs property set -ts "$ROOTVOLMNT/snapshots/$NEWSNAPSHOTS/@" ro true && \ NEWLINKS="$(date -u +"%Y%m%d%H%M%S")$(cat /dev/urandom | tr -dc 'a-zA-Z' | fold -w 8 | head -n 1)" && \ mkdir -p "$ROOTVOLMNT/links/$NEWLINKS" && \ ln -s "../../snapshots/$NEWSNAPSHOTS" "$ROOTVOLMNT/links/$NEWLINKS/0" && \ cp -P "$ROOTVOLMNT/current/0" "$ROOTVOLMNT/links/$NEWLINKS/1" && \ cp -P "$ROOTVOLMNT/current/1" "$ROOTVOLMNT/links/$NEWLINKS/2" && \ cp -P "$ROOTVOLMNT/current/2" "$ROOTVOLMNT/links/$NEWLINKS/3" && \ mkdir -p "$ROOTVOLMNT/next" && \ ln -sfn "./links/$NEWLINKS" "$ROOTVOLMNT/next/current" && \ mv "$ROOTVOLMNT/next/current" "$ROOTVOLMNT/" \ && echo "Switched to the new snapshots" \ || echo "Changes discarded" umount "$ROOTVOLMNT"
It will get you into the root shell chrooted into the new snapshot, where you can apply any change you like.
If chroot shell exits with an error, there will be no switch to the new snapshots. This means you can manually discard changes while in the chroot by:
# exit 1
Deleting unused snapshots
Unused snapshots can be garbage-collected by:
# touch /sbin/syscln # chmod +x /sbin/syscln # vi /sbin/syscln
#!/bin/sh
ROOTVOLMNT="/vol/root"
mkdir -p "$ROOTVOLMNT"
mount -t btrfs -o rw,noatime UUID=b9ff5e7b-e128-4e64-861a-2fdd794a9828 "$ROOTVOLMNT"
SNAPSHOTS=$({ \
find "$ROOTVOLMNT/snapshots/" -maxdepth 1 -mindepth 1
find "$ROOTVOLMNT/current/" -maxdepth 1 -mindepth 1 \
| xargs -n 1 readlink -f
} | sort | uniq -u)
if [ -z "$SNAPSHOTS" ]; then
echo "Nothing to remove"
else
printf %s\\n "$SNAPSHOTS" \
| tr \\n \\0 \
| xargs -r -0 -n 1 -I {} find {} -maxdepth 1 -mindepth 1 \
| tr \\n \\0 \
| xargs -r -0 btrfs subvolume delete
printf %s\\n "$SNAPSHOTS" \
| tr \\n \\0 \
| xargs -r -0 rm -r
fi
umount "$ROOTVOLMNT"
Allowing temporary runtime alterations
You can use overlayfs with tmpfs built into Alpine's init script to allow changes in the rootfs which will be automatically reverted upon reboot.
To make use this just add overlaytmpfs to the kernel boot options in refind.conf, e.g.:
...
initrd /current/0/@/boot/initramfs-lts
options "root=UUID=b9ff5e7b-e128-4e64-861a-2fdd794a9828 rootfstype=btrfs rootflags=subvol=/current/0/@,ro,noatime resume=UUID=f0239163-9d46-47c1-67a4-3ee1d63d0676 overlaytmpfs quiet splash"
submenuentry "Boot fallback 1" {
...