Alpine Linux - User contributions [en]

Root on ZFS with native encryption

2021-01-07T14:19:52Z

R3: del

= Setting up Alpine Linux using ZFS with a pool that uses ZFS' native encryption capabilities =

== Download ==

Download the '''extended''' release from https://www.alpinelinux.org/downloads/ as only it contains the zfs kernel mods at the time of this writing (2020.07.10)

Write it to a USB and boot from it.

== Initial setup ==

Run the following

setup-alpine

Answer all the questions, and hit ctrl-c when promted for what disk you'd like to use.

== OPTIONAL ==

This section is optional and it assumes internet connectivity. You may enable sshd so you can ssh into the box and copy and paste the rest of the commands into my terminal window from these instructions.

Edit `/etc/ssh/sshd_config` and search for `Permit`. Change the value after `PermitRootLogin` to read `yes`

save and exit to shell. Run `service sshd restart`

Now you can ssh in as root. Do not forget to go back and comment this line out when you're done since it will be enabled on your resulting machine. You will be reminded again at the end of this doc.

== Add needed packages ==

apk add zfs sfdisk e2fsprogs syslinux

== Create our partitions ==

We're assuming `/dev/sda` here and in the rest of the document but you can use whatever you need to. To see a list, type: `sfdisk -l`

echo -e "/dev/sda1: start=1M,size=100M,bootable\n/dev/sda2: start=101M" | sfdisk --quiet --label dos /dev/sda

== Create device nodes ==

mdev -s

== Create the /boot filesystem ==

mkfs.ext4 /dev/sda1

== Create the root filesystem using zfs ==

modprobe zfs
zpool create -f -o ashift=12 \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \
-O encryption=aes-256-gcm -O keylocation=prompt -O keyformat=passphrase \
-O mountpoint=/ -R /mnt \
rpool /dev/sda2

You will have to enter your passphrase at this point. Choose wisely, as your passphrase is most likely [https://gitlab.com/cryptsetup/cryptsetup/wikis/FrequentlyAskedQuestions#5-security-aspects the weakest link in this setup].

A few notes on the options supplied to zpool:

- `ashift=12` is recommended here because many drives today have 4KiB (or larger) physical sectors, even though they present 512B logical sectors

- `acltype=posixacl` enables POSIX ACLs globally

- `normalization=formD` eliminates some corner cases relating to UTF-8 filename normalization. It also enables `utf8only=on`, meaning that only files with valid UTF-8 filenames will be accepted.

- `xattr=sa` vastly improves the performance of extended attributes, but is Linux-only. If you care about using this pool on other OpenZFS implementation don't specify this option.

After completing this, confirm that the pool has been created:

# zpool status

Should return something like:

pool: rpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
sda2 ONLINE 0 0 0

errors: No known data errors

== Create the required datasets and mount root ==

zfs create -o mountpoint=none -o canmount=off rpool/ROOT
zfs create -o mountpoint=legacy rpool/ROOT/alpine
mount -t zfs rpool/ROOT/alpine /mnt/

== Mount the `/boot` filesystem ==

mkdir /mnt/boot/
mount -t ext4 /dev/sda1 /mnt/boot/

=== Enable ZFS' services ===

rc-update add zfs-import sysinit
rc-update add zfs-mount sysinit

== Install Alpine Linux ==

setup-disk /mnt
dd if=/usr/share/syslinux/mbr.bin of=/dev/sda # write mbr so we can boot

== Reboot and enjoy! ==

😉

'''NOTE:'''
If you went with the optional step, be sure to disable root login after you reboot.

Root on ZFS with native encryption

2021-01-07T14:13:52Z

R3: rm

This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool <code>/boot</code>, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit <code>-O keylocation -O keyformat</code> when creating root pool.

= Useful links =

*[https://openzfs.github.io/openzfs-docs/Getting%20Started/ OpenZFS Getting Started]

= Notes =

UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

= Preparation =

== Setup live environment ==

Download the '''''extended''''' release from https://www.alpinelinux.org/downloads/, as only this version is shipped with ZFS kernel module. Alpine Linux can not load kernel module in live.

Run the following command to setup the live environment, use default <code>none</code> option when asked about disks.

<pre>setup-alpine</pre>
Settings given here will be copied to the target system later by <code>setup-disk</code>.

== Install system utilities ==

Install and setup<code>eudev</code> (a port of systemd <code>udev</code> by gentoo) to get block device names.

<pre>apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
setup-udev</pre>
= Variables =

In this step, we will set some variables to make our installation process easier.

<pre>DISK=/dev/disk/by-id/ata-HXY_120G_YS</pre>
Use unique disk path instead of <code>/dev/sda</code> to ensure the correct partition can be found by ZFS.

Other variables

<pre>TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'</pre>
Create a mountpoint

<pre>MOUNTPOINT=`mktemp -d`</pre>
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.

<pre>poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)</pre>
= Partitioning =

For a single disk, UEFI installation, we need to create at lease 3 partitions: - EFI system partition - Boot pool partition - Root pool partition Since GRUB only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:

<pre>sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool</pre>
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==

<code>Swap</code> support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:

<pre>sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition</pre>
= Create boot and root pool =

As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no <code>feature@</code> is supplied.

Here we explicitly enable those GRUB can support.

<pre>zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2</pre>
Nothing is stored directly under bpool and rpool, hence <code>canmount=off</code>. The respective <code>mountpoint</code> properties are more symbolic than practical.

For root pool all available features are enabled by default

<pre>echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3</pre>
== For multi-disk ==

For mirror:

<pre>zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3</pre>
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Create system datasets =

This layout is intended to separate root file system from persistent files.

<pre>zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=legacy -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
mkdir $MOUNTPOINT/boot
mount -t zfs bpool_$poolUUID/BOOT/default $MOUNTPOINT/boot
# ash, default with busybox, does not support array
# this is word splitting
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME</pre>
Depending on your application, separate datasets need to be created for folders inside <code>/var/lib</code>(not itself!)

Here we create several folders for persistent (shared) data, like we just did for <code>/home</code>.

<pre>d='libvirt lxc docker'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done</pre>
<code>lxc</code> is for Linux container, <code>libvirt</code> is for storing virtual machine images, etc.

= Format and mount EFI partition =

Here we use <code>/boot/efi</code> as the mountpoint, which is default for GRUB.

<pre>mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi # need to specify file system</pre>
= System installation =

== Preparation ==

GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=1.

<pre>export ZPOOL_VDEV_NAME_PATH=1</pre>
<code>setup-disk</code> refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.

<pre>sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk</pre>
== setup-disk ==

Run <code>setup-disk</code> to install system to target disk.

<pre>BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT</pre>
Note that grub-probe will still fail despite <code>ZPOOL_VDEV_NAME_PATH=YES</code> variable set above. We will deal with this later inside chroot.

== Chroot ==

<pre>m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh</pre>
=== Finish GRUB installation ===

As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply fix:

<pre>echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile</pre>
Reload

<pre>source /etc/profile</pre>
==== GRUB fails to detect the ZFS filesystem of /boot with BusyBox stat ====

<pre>apk add coreutils</pre>
==== Missing root pool ====

GRUB will fail to detect rpool if rpool has unsupported features, use the following workaround:

<pre>sed -i "s|rpool=.*|rpool=\`zdb -l \${GRUB_DEVICE} \| grep -E '[[:blank:]]name' \| cut -d\\\' -f 2\`|" /etc/grub.d/10_linux</pre>
This replaces GRUB rpool name detection.

==== Generate grub.cfg ====

After applying fixes, finally run

<pre>grub-mkconfig -o /boot/grub/grub.cfg</pre>
=== Importing pools on boot ===

<code>zpool.cache</code> will be added to initramfs and zpool command will import pools contained in this cache.

System will fail to boot without this.

<pre>zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID</pre>
=== Initramfs ===

<code>mkinitfs</code> included in stable Alpine Linux has bugs, see [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 1] and [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76 2].

==== Add eudev hook and rebuild ====

Add <code>eudev</code> to <code>/etc/mkinitfs/mkinitfs.conf</code>.

<pre>echo 'features="ata base eudev ide scsi usb virtio nvme zfs"' > /etc/mkinitfs/mkinitfs.conf
# order of features is important! this order is tested</pre>
Rebuild initramfs with

<pre>mkinitfs $(ls -1 /lib/modules/)</pre>

=== Mount datasets at boot ===

<pre>rc-update add zfs-mount sysinit</pre>

=== Add user ===

<pre>adduser -s /bin/sh -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd</pre>
Root account is accessed via <code>su</code> command with root password.

=== Boot environment manager ===

[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request]. Should be available in edge/test soon.

=== Optional: Enable encrypted swap partition ===

Install <code>cryptsetup</code>

<pre>apk add cryptsetup</pre>
Edit the <code>/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the front of zfs. Add relevant lines in <code>fstab</code> and <code>crypttab</code>. Replace <code>$DISK</code> with actual disk.

<pre>echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab</pre>
Rebuild initramfs with <code>mkinitfs</code>.

= Finish installation =

Take a snapshot for the clean installation for future use and export all pools.

<pre>exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install</pre>
Pools must be exported before reboot, or they will fail to be imported on boot.

<pre>mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID</pre>
= Reboot =

<pre>reboot</pre>
= Recovery in Live environment =

Boot Live environment (extended release) and repeat [[#preparation|Preparation]]

Create a mount point and store encryption password in a variable:

<pre>MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'</pre>
Find the unique UUID of your pool with

<pre>zpool import</pre>
Import rpool without mounting datasets: <code>-N</code> for not mounting all datasets; <code>-R</code> for alternate root.

<pre>poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID</pre>
Load encryption key

<pre>echo $ENCRYPTION_PWD | zfs load-key -a</pre>
As <code>canmount=noauto</code> is set for <code>/</code> dataset, we have to mount it manually. To find the dataset, use

<pre>zfs list rpool_$poolUUID/ROOT</pre>
Mount <code>/</code> dataset

<pre>zfs mount rpool_$UUID/ROOT/$dataset</pre>
Mount other datasets

<pre>zfs mount -a</pre>
Import bpool

<pre>zpool import -N -R $MOUNTPOINT bpool_$UUID</pre>
Find and mount the <code>/boot</code> dataset, same as above.

<pre>zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/$dataset $MOUNTPOINT/boot # legacy mountpoint</pre>
Chroot

<pre>mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh</pre>
After chroot, mount <code>/efi</code>

<pre>mount /boot/efi</pre>
After fixing the system, don't forget to umount and export the pools:

<pre>mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID</pre>

Root on ZFS with native encryption

2021-01-07T14:10:52Z

R3: rm

This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool <code>/boot</code>, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit <code>-O keylocation -O keyformat</code> when creating root pool.

= Useful links =

*[https://openzfs.github.io/openzfs-docs/Getting%20Started/ OpenZFS Getting Started]

= Notes =

UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

= Preparation =

== Setup live environment ==

Download the '''''extended''''' release from https://www.alpinelinux.org/downloads/, as only this version is shipped with ZFS kernel module. Alpine Linux can not load kernel module in live.

Run the following command to setup the live environment, use default <code>none</code> option when asked about disks.

<pre>setup-alpine</pre>
Settings given here will be copied to the target system later by <code>setup-disk</code>.

== Install system utilities ==

Install and setup<code>eudev</code> (a port of systemd <code>udev</code> by gentoo) to get block device names.

<pre>apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
setup-udev</pre>
= Variables =

In this step, we will set some variables to make our installation process easier.

<pre>DISK=/dev/disk/by-id/ata-HXY_120G_YS</pre>
Use unique disk path instead of <code>/dev/sda</code> to ensure the correct partition can be found by ZFS.

Other variables

<pre>TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'</pre>
Create a mountpoint

<pre>MOUNTPOINT=`mktemp -d`</pre>
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.

<pre>poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)</pre>
= Partitioning =

For a single disk, UEFI installation, we need to create at lease 3 partitions: - EFI system partition - Boot pool partition - Root pool partition Since GRUB only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:

<pre>sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool</pre>
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==

<code>Swap</code> support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:

<pre>sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition</pre>
= Create boot and root pool =

As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no <code>feature@</code> is supplied.

Here we explicitly enable those GRUB can support.

<pre>zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2</pre>
Nothing is stored directly under bpool and rpool, hence <code>canmount=off</code>. The respective <code>mountpoint</code> properties are more symbolic than practical.

For root pool all available features are enabled by default

<pre>echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3</pre>
== For multi-disk ==

For mirror:

<pre>zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3</pre>
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Create system datasets =

This layout is intended to separate root file system from persistent files. See https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout for a description.

<pre>zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=legacy -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
mkdir $MOUNTPOINT/boot
mount -t zfs bpool_$poolUUID/BOOT/default $MOUNTPOINT/boot
# ash, default with busybox, does not support array
# this is word splitting
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME</pre>
Depending on your application, separate datasets need to be created for folders inside <code>/var/lib</code>(not itself!)

Here we create several folders for persistent (shared) data, like we just did for <code>/home</code>.

<pre>d='libvirt lxc docker'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done</pre>
<code>lxc</code> is for Linux container, <code>libvirt</code> is for storing virtual machine images, etc.

= Format and mount EFI partition =

Here we use <code>/boot/efi</code> as the mountpoint, which is default for GRUB.

<pre>mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi # need to specify file system</pre>
= System installation =

== Preparation ==

GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=1.

<pre>export ZPOOL_VDEV_NAME_PATH=1</pre>
<code>setup-disk</code> refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.

<pre>sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk</pre>
== setup-disk ==

Run <code>setup-disk</code> to install system to target disk.

<pre>BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT</pre>
Note that grub-probe will still fail despite <code>ZPOOL_VDEV_NAME_PATH=YES</code> variable set above. We will deal with this later inside chroot.

== Chroot ==

<pre>m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh</pre>
=== Finish GRUB installation ===

As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply fix:

<pre>echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile</pre>
Reload

<pre>source /etc/profile</pre>
==== GRUB fails to detect the ZFS filesystem of /boot with BusyBox stat ====

<pre>apk add coreutils</pre>
==== Missing root pool ====

GRUB will fail to detect rpool if rpool has unsupported features, use the following workaround:

<pre>sed -i "s|rpool=.*|rpool=\`zdb -l \${GRUB_DEVICE} \| grep -E '[[:blank:]]name' \| cut -d\\\' -f 2\`|" /etc/grub.d/10_linux</pre>
This replaces GRUB rpool name detection.

==== Generate grub.cfg ====

After applying fixes, finally run

<pre>grub-mkconfig -o /boot/grub/grub.cfg</pre>
=== Importing pools on boot ===

<code>zpool.cache</code> will be added to initramfs and zpool command will import pools contained in this cache.

System will fail to boot without this.

<pre>zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID</pre>
=== Initramfs ===

<code>mkinitfs</code> included in stable Alpine Linux has bugs, see [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 1] and [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76 2].

==== Add eudev hook and rebuild ====

Add <code>eudev</code> to <code>/etc/mkinitfs/mkinitfs.conf</code>.

<pre>echo 'features="ata base eudev ide scsi usb virtio nvme zfs"' > /etc/mkinitfs/mkinitfs.conf
# order of features is important! this order is tested</pre>
Rebuild initramfs with

<pre>mkinitfs $(ls -1 /lib/modules/)</pre>

=== Mount datasets at boot ===

<pre>rc-update add zfs-mount sysinit</pre>

=== Add user ===

<pre>adduser -s /bin/sh -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd</pre>
Root account is accessed via <code>su</code> command with root password.

=== Boot environment manager ===

[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request]. Should be available in edge/test soon.

=== Optional: Enable encrypted swap partition ===

Install <code>cryptsetup</code>

<pre>apk add cryptsetup</pre>
Edit the <code>/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the front of zfs. Add relevant lines in <code>fstab</code> and <code>crypttab</code>. Replace <code>$DISK</code> with actual disk.

<pre>echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab</pre>
Rebuild initramfs with <code>mkinitfs</code>.

= Finish installation =

Take a snapshot for the clean installation for future use and export all pools.

<pre>exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install</pre>
Pools must be exported before reboot, or they will fail to be imported on boot.

<pre>mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID</pre>
= Reboot =

<pre>reboot</pre>
= Recovery in Live environment =

Boot Live environment (extended release) and repeat [[#preparation|Preparation]]

Create a mount point and store encryption password in a variable:

<pre>MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'</pre>
Find the unique UUID of your pool with

<pre>zpool import</pre>
Import rpool without mounting datasets: <code>-N</code> for not mounting all datasets; <code>-R</code> for alternate root.

<pre>poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID</pre>
Load encryption key

<pre>echo $ENCRYPTION_PWD | zfs load-key -a</pre>
As <code>canmount=noauto</code> is set for <code>/</code> dataset, we have to mount it manually. To find the dataset, use

<pre>zfs list rpool_$poolUUID/ROOT</pre>
Mount <code>/</code> dataset

<pre>zfs mount rpool_$UUID/ROOT/$dataset</pre>
Mount other datasets

<pre>zfs mount -a</pre>
Import bpool

<pre>zpool import -N -R $MOUNTPOINT bpool_$UUID</pre>
Find and mount the <code>/boot</code> dataset, same as above.

<pre>zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/$dataset $MOUNTPOINT/boot # legacy mountpoint</pre>
Chroot

<pre>mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh</pre>
After chroot, mount <code>/efi</code>

<pre>mount /boot/efi</pre>
After fixing the system, don't forget to umount and export the pools:

<pre>mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID</pre>

Root on ZFS with native encryption

2021-01-07T08:40:57Z

R3: /* Initramfs */ fix link

This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool <code>/boot</code>, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit <code>-O keylocation -O keyformat</code> when creating root pool.

= Useful links =

*[https://openzfs.github.io/openzfs-docs/Getting%20Started/ OpenZFS Getting Started]
*[https://g.nu8.org/posts/bieaz/setup/alpine/guide/ Encrypted ZFS with boot environment support]

= Notes =

UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

= Preparation =

== Setup live environment ==

Download the '''''extended''''' release from https://www.alpinelinux.org/downloads/, as only this version is shipped with ZFS kernel module. Alpine Linux can not load kernel module in live.

Run the following command to setup the live environment, use default <code>none</code> option when asked about disks.

<pre>setup-alpine</pre>
Settings given here will be copied to the target system later by <code>setup-disk</code>.

== Install system utilities ==

Install and setup<code>eudev</code> (a port of systemd <code>udev</code> by gentoo) to get block device names.

<pre>apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
setup-udev</pre>
= Variables =

In this step, we will set some variables to make our installation process easier.

<pre>DISK=/dev/disk/by-id/ata-HXY_120G_YS</pre>
Use unique disk path instead of <code>/dev/sda</code> to ensure the correct partition can be found by ZFS.

Other variables

<pre>TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'</pre>
Create a mountpoint

<pre>MOUNTPOINT=`mktemp -d`</pre>
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.

<pre>poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)</pre>
= Partitioning =

For a single disk, UEFI installation, we need to create at lease 3 partitions: - EFI system partition - Boot pool partition - Root pool partition Since GRUB only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:

<pre>sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool</pre>
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==

<code>Swap</code> support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:

<pre>sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition</pre>
= Create boot and root pool =

As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no <code>feature@</code> is supplied.

Here we explicitly enable those GRUB can support.

<pre>zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2</pre>
Nothing is stored directly under bpool and rpool, hence <code>canmount=off</code>. The respective <code>mountpoint</code> properties are more symbolic than practical.

For root pool all available features are enabled by default

<pre>echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3</pre>
== For multi-disk ==

For mirror:

<pre>zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3</pre>
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Create system datasets =

This layout is intended to separate root file system from persistent files. See https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout for a description.

<pre>zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=legacy -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
mkdir $MOUNTPOINT/boot
mount -t zfs bpool_$poolUUID/BOOT/default $MOUNTPOINT/boot
# ash, default with busybox, does not support array
# this is word splitting
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME</pre>
Depending on your application, separate datasets need to be created for folders inside <code>/var/lib</code>(not itself!)

Here we create several folders for persistent (shared) data, like we just did for <code>/home</code>.

<pre>d='libvirt lxc docker'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done</pre>
<code>lxc</code> is for Linux container, <code>libvirt</code> is for storing virtual machine images, etc.

= Format and mount EFI partition =

Here we use <code>/boot/efi</code> as the mountpoint, which is default for GRUB.

<pre>mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi # need to specify file system</pre>
= System installation =

== Preparation ==

GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=1.

<pre>export ZPOOL_VDEV_NAME_PATH=1</pre>
<code>setup-disk</code> refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.

<pre>sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk</pre>
== setup-disk ==

Run <code>setup-disk</code> to install system to target disk.

<pre>BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT</pre>
Note that grub-probe will still fail despite <code>ZPOOL_VDEV_NAME_PATH=YES</code> variable set above. We will deal with this later inside chroot.

== Chroot ==

<pre>m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh</pre>
=== Finish GRUB installation ===

As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply fix:

<pre>echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile</pre>
Reload

<pre>source /etc/profile</pre>
==== GRUB fails to detect the ZFS filesystem of /boot with BusyBox stat ====

<pre>apk add coreutils</pre>
==== Missing root pool ====

GRUB will fail to detect rpool if rpool has unsupported features, use the following workaround:

<pre>sed -i "s|rpool=.*|rpool=\`zdb -l \${GRUB_DEVICE} \| grep -E '[[:blank:]]name' \| cut -d\\\' -f 2\`|" /etc/grub.d/10_linux</pre>
This replaces GRUB rpool name detection.

==== Generate grub.cfg ====

After applying fixes, finally run

<pre>grub-mkconfig -o /boot/grub/grub.cfg</pre>
=== Importing pools on boot ===

<code>zpool.cache</code> will be added to initramfs and zpool command will import pools contained in this cache.

System will fail to boot without this.

<pre>zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID</pre>
=== Initramfs ===

<code>mkinitfs</code> included in stable Alpine Linux has bugs, before [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 1] and [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76 2] is merged, we need to patch it manually. #### Patch Ensure <code>mkinitfs</code> version is the following

<pre>foolive:/# apk info mkinitfs
mkinitfs-3.4.5-r3 description:</pre>
Then download [https://g.nu8.org/posts/bieaz/setup/alpine/guide/patch/eudev-zfs-mkinitfs-3.4.5.patch eudev-zfs-mkinitfs-3.4.5.patch], install <code>patch</code> and patch it.

<pre>foolive:~# wget https://g.nu8.org/path-to-patch
foolive:~# apk add patch
foolive:~# cd / # must apply patch at root
foolive:/# patch -Np1 -i /root/eudev-zfs-mkinitfs-3.4.5.patch
patching file etc/mkinitfs/features.d/eudev.files
patching file etc/mkinitfs/features.d/zfs.files
patching file usr/share/mkinitfs/initramfs-init</pre>
==== Add eudev hook and rebuild ====

Add <code>eudev</code> to <code>/etc/mkinitfs/mkinitfs.conf</code>.

<pre>echo 'features="ata base eudev ide scsi usb virtio nvme zfs"' > /etc/mkinitfs/mkinitfs.conf
# order of features is important! this order is tested</pre>
Rebuild initramfs with

<pre>mkinitfs $(ls -1 /lib/modules/)</pre>

=== Mount datasets at boot ===

<pre>rc-update add zfs-mount sysinit</pre>

=== Add user ===

<pre>adduser -s /bin/sh -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd</pre>
Root account is accessed via <code>su</code> command with root password.

=== Boot environment manager ===

[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request]. Should be available in edge/test soon.

=== Optional: Enable encrypted swap partition ===

Install <code>cryptsetup</code>

<pre>apk add cryptsetup</pre>
Edit the <code>/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the front of zfs. Add relevant lines in <code>fstab</code> and <code>crypttab</code>. Replace <code>$DISK</code> with actual disk.

<pre>echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab</pre>
Rebuild initramfs with <code>mkinitfs</code>.

= Finish installation =

Take a snapshot for the clean installation for future use and export all pools.

<pre>exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install</pre>
Pools must be exported before reboot, or they will fail to be imported on boot.

<pre>mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID</pre>
= Reboot =

<pre>reboot</pre>
= Recovery in Live environment =

Boot Live environment (extended release) and repeat [[#preparation|Preparation]]

Create a mount point and store encryption password in a variable:

<pre>MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'</pre>
Find the unique UUID of your pool with

<pre>zpool import</pre>
Import rpool without mounting datasets: <code>-N</code> for not mounting all datasets; <code>-R</code> for alternate root.

<pre>poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID</pre>
Load encryption key

<pre>echo $ENCRYPTION_PWD | zfs load-key -a</pre>
As <code>canmount=noauto</code> is set for <code>/</code> dataset, we have to mount it manually. To find the dataset, use

<pre>zfs list rpool_$poolUUID/ROOT</pre>
Mount <code>/</code> dataset

<pre>zfs mount rpool_$UUID/ROOT/$dataset</pre>
Mount other datasets

<pre>zfs mount -a</pre>
Import bpool

<pre>zpool import -N -R $MOUNTPOINT bpool_$UUID</pre>
Find and mount the <code>/boot</code> dataset, same as above.

<pre>zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/$dataset $MOUNTPOINT/boot # legacy mountpoint</pre>
Chroot

<pre>mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh</pre>
After chroot, mount <code>/efi</code>

<pre>mount /boot/efi</pre>
After fixing the system, don't forget to umount and export the pools:

<pre>mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID</pre>

Talk:Alpine Linux with root on ZFS with native encryption

2021-01-07T00:25:29Z

R3: R3 moved page Talk:Alpine Linux with root on ZFS with native encryption to Talk:Root on ZFS with native encryption: redundant alpine linux in title

#REDIRECT [[Talk:Root on ZFS with native encryption]]

Talk:Root on ZFS with native encryption

2021-01-07T00:25:29Z

R3: R3 moved page Talk:Alpine Linux with root on ZFS with native encryption to Talk:Root on ZFS with native encryption: redundant alpine linux in title

= Alpine on ZFS root issues in wiki procedure (v1) =

I made some notes on issues I have encountered while following this guide. I will check these more and see if I can update the wiki with the notes.

You can find the notes here: [https://pastebin.com/7jXtG6pT Notes on pastebin]

~~

Alpine Linux with root on ZFS with native encryption

2021-01-07T00:25:29Z

R3: R3 moved page Alpine Linux with root on ZFS with native encryption to Root on ZFS with native encryption: redundant alpine linux in title

#REDIRECT [[Root on ZFS with native encryption]]

Root on ZFS with native encryption

2021-01-07T00:25:29Z

R3: R3 moved page Alpine Linux with root on ZFS with native encryption to Root on ZFS with native encryption: redundant alpine linux in title

This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool <code>/boot</code>, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit <code>-O keylocation -O keyformat</code> when creating root pool.

= Useful links =

*[https://openzfs.github.io/openzfs-docs/Getting%20Started/ OpenZFS Getting Started]
*[https://g.nu8.org/posts/bieaz/setup/alpine/guide/ Encrypted ZFS with boot environment support]

= Notes =

UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

= Preparation =

== Setup live environment ==

Download the '''''extended''''' release from https://www.alpinelinux.org/downloads/, as only this version is shipped with ZFS kernel module. Alpine Linux can not load kernel module in live.

Run the following command to setup the live environment, use default <code>none</code> option when asked about disks.

<pre>setup-alpine</pre>
Settings given here will be copied to the target system later by <code>setup-disk</code>.

== Install system utilities ==

Install and setup<code>eudev</code> (a port of systemd <code>udev</code> by gentoo) to get block device names.

<pre>apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
setup-udev</pre>
= Variables =

In this step, we will set some variables to make our installation process easier.

<pre>DISK=/dev/disk/by-id/ata-HXY_120G_YS</pre>
Use unique disk path instead of <code>/dev/sda</code> to ensure the correct partition can be found by ZFS.

Other variables

<pre>TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'</pre>
Create a mountpoint

<pre>MOUNTPOINT=`mktemp -d`</pre>
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.

<pre>poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)</pre>
= Partitioning =

For a single disk, UEFI installation, we need to create at lease 3 partitions: - EFI system partition - Boot pool partition - Root pool partition Since GRUB only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:

<pre>sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool</pre>
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==

<code>Swap</code> support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:

<pre>sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition</pre>
= Create boot and root pool =

As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no <code>feature@</code> is supplied.

Here we explicitly enable those GRUB can support.

<pre>zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2</pre>
Nothing is stored directly under bpool and rpool, hence <code>canmount=off</code>. The respective <code>mountpoint</code> properties are more symbolic than practical.

For root pool all available features are enabled by default

<pre>echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3</pre>
== For multi-disk ==

For mirror:

<pre>zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3</pre>
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Create system datasets =

This layout is intended to separate root file system from persistent files. See https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout for a description.

<pre>zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=legacy -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
mkdir $MOUNTPOINT/boot
mount -t zfs bpool_$poolUUID/BOOT/default $MOUNTPOINT/boot
# ash, default with busybox, does not support array
# this is word splitting
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME</pre>
Depending on your application, separate datasets need to be created for folders inside <code>/var/lib</code>(not itself!)

Here we create several folders for persistent (shared) data, like we just did for <code>/home</code>.

<pre>d='libvirt lxc docker'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done</pre>
<code>lxc</code> is for Linux container, <code>libvirt</code> is for storing virtual machine images, etc.

= Format and mount EFI partition =

Here we use <code>/boot/efi</code> as the mountpoint, which is default for GRUB.

<pre>mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi # need to specify file system</pre>
= System installation =

== Preparation ==

GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=1.

<pre>export ZPOOL_VDEV_NAME_PATH=1</pre>
<code>setup-disk</code> refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.

<pre>sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk</pre>
== setup-disk ==

Run <code>setup-disk</code> to install system to target disk.

<pre>BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT</pre>
Note that grub-probe will still fail despite <code>ZPOOL_VDEV_NAME_PATH=YES</code> variable set above. We will deal with this later inside chroot.

== Chroot ==

<pre>m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh</pre>
=== Finish GRUB installation ===

As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply fix:

<pre>echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile</pre>
Reload

<pre>source /etc/profile</pre>
==== GRUB fails to detect the ZFS filesystem of /boot with BusyBox stat ====

<pre>apk add coreutils</pre>
==== Missing root pool ====

GRUB will fail to detect rpool if rpool has unsupported features, use the following workaround:

<pre>sed -i "s|rpool=.*|rpool=\`zdb -l \${GRUB_DEVICE} \| grep -E '[[:blank:]]name' \| cut -d\\\' -f 2\`|" /etc/grub.d/10_linux</pre>
This replaces GRUB rpool name detection.

==== Generate grub.cfg ====

After applying fixes, finally run

<pre>grub-mkconfig -o /boot/grub/grub.cfg</pre>
=== Importing pools on boot ===

<code>zpool.cache</code> will be added to initramfs and zpool command will import pools contained in this cache.

System will fail to boot without this.

<pre>zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID</pre>
=== Initramfs ===

<code>mkinitfs</code> included in stable Alpine Linux has bugs, before [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 1] and [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76 2] is merged, we need to patch it manually. #### Patch Ensure <code>mkinitfs</code> version is the following

<pre>foolive:/# apk info mkinitfs
mkinitfs-3.4.5-r3 description:</pre>
Then download [https://g.nu8.org/posts/bieaz/setup/alpine/guide/patch/eudev-zfs-mkinitfs-3.4.5.patch|eudev-zfs-mkinitfs-3.4.5.patch], install <code>patch</code> and patch it.

<pre>foolive:~# wget https://g.nu8.org/path-to-patch
foolive:~# apk add patch
foolive:~# cd / # must apply patch at root
foolive:/# patch -Np1 -i /root/eudev-zfs-mkinitfs-3.4.5.patch
patching file etc/mkinitfs/features.d/eudev.files
patching file etc/mkinitfs/features.d/zfs.files
patching file usr/share/mkinitfs/initramfs-init</pre>
==== Add eudev hook and rebuild ====

Add <code>eudev</code> to <code>/etc/mkinitfs/mkinitfs.conf</code>.

<pre>echo 'features="ata base eudev ide scsi usb virtio nvme zfs"' > /etc/mkinitfs/mkinitfs.conf
# order of features is important! this order is tested</pre>
Rebuild initramfs with

<pre>mkinitfs $(ls -1 /lib/modules/)</pre>

=== Mount datasets at boot ===

<pre>rc-update add zfs-mount sysinit</pre>

=== Add user ===

<pre>adduser -s /bin/sh -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd</pre>
Root account is accessed via <code>su</code> command with root password.

=== Boot environment manager ===

[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request]. Should be available in edge/test soon.

=== Optional: Enable encrypted swap partition ===

Install <code>cryptsetup</code>

<pre>apk add cryptsetup</pre>
Edit the <code>/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the front of zfs. Add relevant lines in <code>fstab</code> and <code>crypttab</code>. Replace <code>$DISK</code> with actual disk.

<pre>echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab</pre>
Rebuild initramfs with <code>mkinitfs</code>.

= Finish installation =

Take a snapshot for the clean installation for future use and export all pools.

<pre>exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install</pre>
Pools must be exported before reboot, or they will fail to be imported on boot.

<pre>mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID</pre>
= Reboot =

<pre>reboot</pre>
= Recovery in Live environment =

Boot Live environment (extended release) and repeat [[#preparation|Preparation]]

Create a mount point and store encryption password in a variable:

<pre>MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'</pre>
Find the unique UUID of your pool with

<pre>zpool import</pre>
Import rpool without mounting datasets: <code>-N</code> for not mounting all datasets; <code>-R</code> for alternate root.

<pre>poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID</pre>
Load encryption key

<pre>echo $ENCRYPTION_PWD | zfs load-key -a</pre>
As <code>canmount=noauto</code> is set for <code>/</code> dataset, we have to mount it manually. To find the dataset, use

<pre>zfs list rpool_$poolUUID/ROOT</pre>
Mount <code>/</code> dataset

<pre>zfs mount rpool_$UUID/ROOT/$dataset</pre>
Mount other datasets

<pre>zfs mount -a</pre>
Import bpool

<pre>zpool import -N -R $MOUNTPOINT bpool_$UUID</pre>
Find and mount the <code>/boot</code> dataset, same as above.

<pre>zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/$dataset $MOUNTPOINT/boot # legacy mountpoint</pre>
Chroot

<pre>mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh</pre>
After chroot, mount <code>/efi</code>

<pre>mount /boot/efi</pre>
After fixing the system, don't forget to umount and export the pools:

<pre>mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID</pre>

Root on ZFS with native encryption

2021-01-07T00:24:25Z

R3: links

This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool <code>/boot</code>, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit <code>-O keylocation -O keyformat</code> when creating root pool.

= Useful links =

*[https://openzfs.github.io/openzfs-docs/Getting%20Started/ OpenZFS Getting Started]
*[https://g.nu8.org/posts/bieaz/setup/alpine/guide/ Encrypted ZFS with boot environment support]

= Notes =

UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

= Preparation =

== Setup live environment ==

Download the '''''extended''''' release from https://www.alpinelinux.org/downloads/, as only this version is shipped with ZFS kernel module. Alpine Linux can not load kernel module in live.

Run the following command to setup the live environment, use default <code>none</code> option when asked about disks.

<pre>setup-alpine</pre>
Settings given here will be copied to the target system later by <code>setup-disk</code>.

== Install system utilities ==

Install and setup<code>eudev</code> (a port of systemd <code>udev</code> by gentoo) to get block device names.

<pre>apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
setup-udev</pre>
= Variables =

In this step, we will set some variables to make our installation process easier.

<pre>DISK=/dev/disk/by-id/ata-HXY_120G_YS</pre>
Use unique disk path instead of <code>/dev/sda</code> to ensure the correct partition can be found by ZFS.

Other variables

<pre>TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'</pre>
Create a mountpoint

<pre>MOUNTPOINT=`mktemp -d`</pre>
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.

<pre>poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)</pre>
= Partitioning =

For a single disk, UEFI installation, we need to create at lease 3 partitions: - EFI system partition - Boot pool partition - Root pool partition Since GRUB only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:

<pre>sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool</pre>
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==

<code>Swap</code> support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:

<pre>sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition</pre>
= Create boot and root pool =

As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no <code>feature@</code> is supplied.

Here we explicitly enable those GRUB can support.

<pre>zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2</pre>
Nothing is stored directly under bpool and rpool, hence <code>canmount=off</code>. The respective <code>mountpoint</code> properties are more symbolic than practical.

For root pool all available features are enabled by default

<pre>echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3</pre>
== For multi-disk ==

For mirror:

<pre>zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3</pre>
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Create system datasets =

This layout is intended to separate root file system from persistent files. See https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout for a description.

<pre>zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=legacy -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
mkdir $MOUNTPOINT/boot
mount -t zfs bpool_$poolUUID/BOOT/default $MOUNTPOINT/boot
# ash, default with busybox, does not support array
# this is word splitting
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME</pre>
Depending on your application, separate datasets need to be created for folders inside <code>/var/lib</code>(not itself!)

Here we create several folders for persistent (shared) data, like we just did for <code>/home</code>.

<pre>d='libvirt lxc docker'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done</pre>
<code>lxc</code> is for Linux container, <code>libvirt</code> is for storing virtual machine images, etc.

= Format and mount EFI partition =

Here we use <code>/boot/efi</code> as the mountpoint, which is default for GRUB.

<pre>mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi # need to specify file system</pre>
= System installation =

== Preparation ==

GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=1.

<pre>export ZPOOL_VDEV_NAME_PATH=1</pre>
<code>setup-disk</code> refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.

<pre>sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk</pre>
== setup-disk ==

Run <code>setup-disk</code> to install system to target disk.

<pre>BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT</pre>
Note that grub-probe will still fail despite <code>ZPOOL_VDEV_NAME_PATH=YES</code> variable set above. We will deal with this later inside chroot.

== Chroot ==

<pre>m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh</pre>
=== Finish GRUB installation ===

As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply fix:

<pre>echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile</pre>
Reload

<pre>source /etc/profile</pre>
==== GRUB fails to detect the ZFS filesystem of /boot with BusyBox stat ====

<pre>apk add coreutils</pre>
==== Missing root pool ====

GRUB will fail to detect rpool if rpool has unsupported features, use the following workaround:

<pre>sed -i "s|rpool=.*|rpool=\`zdb -l \${GRUB_DEVICE} \| grep -E '[[:blank:]]name' \| cut -d\\\' -f 2\`|" /etc/grub.d/10_linux</pre>
This replaces GRUB rpool name detection.

==== Generate grub.cfg ====

After applying fixes, finally run

<pre>grub-mkconfig -o /boot/grub/grub.cfg</pre>
=== Importing pools on boot ===

<code>zpool.cache</code> will be added to initramfs and zpool command will import pools contained in this cache.

System will fail to boot without this.

<pre>zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID</pre>
=== Initramfs ===

<code>mkinitfs</code> included in stable Alpine Linux has bugs, before [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 1] and [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76 2] is merged, we need to patch it manually. #### Patch Ensure <code>mkinitfs</code> version is the following

<pre>foolive:/# apk info mkinitfs
mkinitfs-3.4.5-r3 description:</pre>
Then download [https://g.nu8.org/posts/bieaz/setup/alpine/guide/patch/eudev-zfs-mkinitfs-3.4.5.patch|eudev-zfs-mkinitfs-3.4.5.patch], install <code>patch</code> and patch it.

<pre>foolive:~# wget https://g.nu8.org/path-to-patch
foolive:~# apk add patch
foolive:~# cd / # must apply patch at root
foolive:/# patch -Np1 -i /root/eudev-zfs-mkinitfs-3.4.5.patch
patching file etc/mkinitfs/features.d/eudev.files
patching file etc/mkinitfs/features.d/zfs.files
patching file usr/share/mkinitfs/initramfs-init</pre>
==== Add eudev hook and rebuild ====

Add <code>eudev</code> to <code>/etc/mkinitfs/mkinitfs.conf</code>.

<pre>echo 'features="ata base eudev ide scsi usb virtio nvme zfs"' > /etc/mkinitfs/mkinitfs.conf
# order of features is important! this order is tested</pre>
Rebuild initramfs with

<pre>mkinitfs $(ls -1 /lib/modules/)</pre>

=== Mount datasets at boot ===

<pre>rc-update add zfs-mount sysinit</pre>

=== Add user ===

<pre>adduser -s /bin/sh -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd</pre>
Root account is accessed via <code>su</code> command with root password.

=== Boot environment manager ===

[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request]. Should be available in edge/test soon.

=== Optional: Enable encrypted swap partition ===

Install <code>cryptsetup</code>

<pre>apk add cryptsetup</pre>
Edit the <code>/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the front of zfs. Add relevant lines in <code>fstab</code> and <code>crypttab</code>. Replace <code>$DISK</code> with actual disk.

<pre>echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab</pre>
Rebuild initramfs with <code>mkinitfs</code>.

= Finish installation =

Take a snapshot for the clean installation for future use and export all pools.

<pre>exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install</pre>
Pools must be exported before reboot, or they will fail to be imported on boot.

<pre>mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID</pre>
= Reboot =

<pre>reboot</pre>
= Recovery in Live environment =

Boot Live environment (extended release) and repeat [[#preparation|Preparation]]

Create a mount point and store encryption password in a variable:

<pre>MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'</pre>
Find the unique UUID of your pool with

<pre>zpool import</pre>
Import rpool without mounting datasets: <code>-N</code> for not mounting all datasets; <code>-R</code> for alternate root.

<pre>poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID</pre>
Load encryption key

<pre>echo $ENCRYPTION_PWD | zfs load-key -a</pre>
As <code>canmount=noauto</code> is set for <code>/</code> dataset, we have to mount it manually. To find the dataset, use

<pre>zfs list rpool_$poolUUID/ROOT</pre>
Mount <code>/</code> dataset

<pre>zfs mount rpool_$UUID/ROOT/$dataset</pre>
Mount other datasets

<pre>zfs mount -a</pre>
Import bpool

<pre>zpool import -N -R $MOUNTPOINT bpool_$UUID</pre>
Find and mount the <code>/boot</code> dataset, same as above.

<pre>zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/$dataset $MOUNTPOINT/boot # legacy mountpoint</pre>
Chroot

<pre>mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh</pre>
After chroot, mount <code>/efi</code>

<pre>mount /boot/efi</pre>
After fixing the system, don't forget to umount and export the pools:

<pre>mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID</pre>

Root on ZFS with native encryption

2021-01-07T00:05:47Z

R3: /* Mount datasets at boot */ rm

This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool <code>/boot</code>, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit <code>-O keylocation -O keyformat</code> when creating root pool.

= Notes =

UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

= Preparation =

== Setup live environment ==

Download the '''''extended''''' release from https://www.alpinelinux.org/downloads/, as only this version is shipped with ZFS kernel module. Alpine Linux can not load kernel module in live.

Run the following command to setup the live environment, use default <code>none</code> option when asked about disks.

<pre>setup-alpine</pre>
Settings given here will be copied to the target system later by <code>setup-disk</code>.

== Install system utilities ==

Install and setup<code>eudev</code> (a port of systemd <code>udev</code> by gentoo) to get block device names.

<pre>apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
setup-udev</pre>
= Variables =

In this step, we will set some variables to make our installation process easier.

<pre>DISK=/dev/disk/by-id/ata-HXY_120G_YS</pre>
Use unique disk path instead of <code>/dev/sda</code> to ensure the correct partition can be found by ZFS.

Other variables

<pre>TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'</pre>
Create a mountpoint

<pre>MOUNTPOINT=`mktemp -d`</pre>
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.

<pre>poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)</pre>
= Partitioning =

For a single disk, UEFI installation, we need to create at lease 3 partitions: - EFI system partition - Boot pool partition - Root pool partition Since GRUB only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:

<pre>sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool</pre>
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==

<code>Swap</code> support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:

<pre>sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition</pre>
= Create boot and root pool =

As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no <code>feature@</code> is supplied.

Here we explicitly enable those GRUB can support.

<pre>zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2</pre>
Nothing is stored directly under bpool and rpool, hence <code>canmount=off</code>. The respective <code>mountpoint</code> properties are more symbolic than practical.

For root pool all available features are enabled by default

<pre>echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3</pre>
== For multi-disk ==

For mirror:

<pre>zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3</pre>
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Create system datasets =

This layout is intended to separate root file system from persistent files. See https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout for a description.

<pre>zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=legacy -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
mkdir $MOUNTPOINT/boot
mount -t zfs bpool_$poolUUID/BOOT/default $MOUNTPOINT/boot
# ash, default with busybox, does not support array
# this is word splitting
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME</pre>
Depending on your application, separate datasets need to be created for folders inside <code>/var/lib</code>(not itself!)

Here we create several folders for persistent (shared) data, like we just did for <code>/home</code>.

<pre>d='libvirt lxc docker'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done</pre>
<code>lxc</code> is for Linux container, <code>libvirt</code> is for storing virtual machine images, etc.

= Format and mount EFI partition =

Here we use <code>/boot/efi</code> as the mountpoint, which is default for GRUB.

<pre>mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi # need to specify file system</pre>
= System installation =

== Preparation ==

GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=1.

<pre>export ZPOOL_VDEV_NAME_PATH=1</pre>
<code>setup-disk</code> refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.

<pre>sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk</pre>
== setup-disk ==

Run <code>setup-disk</code> to install system to target disk.

<pre>BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT</pre>
Note that grub-probe will still fail despite <code>ZPOOL_VDEV_NAME_PATH=YES</code> variable set above. We will deal with this later inside chroot.

== Chroot ==

<pre>m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh</pre>
=== Finish GRUB installation ===

As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply fix:

<pre>echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile</pre>
Reload

<pre>source /etc/profile</pre>
==== GRUB fails to detect the ZFS filesystem of /boot with BusyBox stat ====

<pre>apk add coreutils</pre>
==== Missing root pool ====

GRUB will fail to detect rpool if rpool has unsupported features, use the following workaround:

<pre>sed -i "s|rpool=.*|rpool=\`zdb -l \${GRUB_DEVICE} \| grep -E '[[:blank:]]name' \| cut -d\\\' -f 2\`|" /etc/grub.d/10_linux</pre>
This replaces GRUB rpool name detection.

==== Generate grub.cfg ====

After applying fixes, finally run

<pre>grub-mkconfig -o /boot/grub/grub.cfg</pre>
=== Importing pools on boot ===

<code>zpool.cache</code> will be added to initramfs and zpool command will import pools contained in this cache.

System will fail to boot without this.

<pre>zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID</pre>
=== Initramfs ===

<code>mkinitfs</code> included in stable Alpine Linux has bugs, before [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 1] and [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76 2] is merged, we need to patch it manually. #### Patch Ensure <code>mkinitfs</code> version is the following

<pre>foolive:/# apk info mkinitfs
mkinitfs-3.4.5-r3 description:</pre>
Then download [https://g.nu8.org/posts/bieaz/setup/alpine/guide/patch/eudev-zfs-mkinitfs-3.4.5.patch|eudev-zfs-mkinitfs-3.4.5.patch], install <code>patch</code> and patch it.

<pre>foolive:~# wget https://g.nu8.org/path-to-patch
foolive:~# apk add patch
foolive:~# cd / # must apply patch at root
foolive:/# patch -Np1 -i /root/eudev-zfs-mkinitfs-3.4.5.patch
patching file etc/mkinitfs/features.d/eudev.files
patching file etc/mkinitfs/features.d/zfs.files
patching file usr/share/mkinitfs/initramfs-init</pre>
==== Add eudev hook and rebuild ====

Add <code>eudev</code> to <code>/etc/mkinitfs/mkinitfs.conf</code>.

<pre>echo 'features="ata base eudev ide scsi usb virtio nvme zfs"' > /etc/mkinitfs/mkinitfs.conf
# order of features is important! this order is tested</pre>
Rebuild initramfs with

<pre>mkinitfs $(ls -1 /lib/modules/)</pre>

=== Mount datasets at boot ===

<pre>rc-update add zfs-mount sysinit</pre>

=== Add user ===

<pre>adduser -s /bin/sh -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd</pre>
Root account is accessed via <code>su</code> command with root password.

=== Boot environment manager ===

[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request]. Should be available in edge/test soon.

=== Optional: Enable encrypted swap partition ===

Install <code>cryptsetup</code>

<pre>apk add cryptsetup</pre>
Edit the <code>/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the front of zfs. Add relevant lines in <code>fstab</code> and <code>crypttab</code>. Replace <code>$DISK</code> with actual disk.

<pre>echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab</pre>
Rebuild initramfs with <code>mkinitfs</code>.

= Finish installation =

Take a snapshot for the clean installation for future use and export all pools.

<pre>exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install</pre>
Pools must be exported before reboot, or they will fail to be imported on boot.

<pre>mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID</pre>
= Reboot =

<pre>reboot</pre>
= Recovery in Live environment =

Boot Live environment (extended release) and repeat [[#preparation|Preparation]]

Create a mount point and store encryption password in a variable:

<pre>MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'</pre>
Find the unique UUID of your pool with

<pre>zpool import</pre>
Import rpool without mounting datasets: <code>-N</code> for not mounting all datasets; <code>-R</code> for alternate root.

<pre>poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID</pre>
Load encryption key

<pre>echo $ENCRYPTION_PWD | zfs load-key -a</pre>
As <code>canmount=noauto</code> is set for <code>/</code> dataset, we have to mount it manually. To find the dataset, use

<pre>zfs list rpool_$poolUUID/ROOT</pre>
Mount <code>/</code> dataset

<pre>zfs mount rpool_$UUID/ROOT/$dataset</pre>
Mount other datasets

<pre>zfs mount -a</pre>
Import bpool

<pre>zpool import -N -R $MOUNTPOINT bpool_$UUID</pre>
Find and mount the <code>/boot</code> dataset, same as above.

<pre>zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/$dataset $MOUNTPOINT/boot # legacy mountpoint</pre>
Chroot

<pre>mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh</pre>
After chroot, mount <code>/efi</code>

<pre>mount /boot/efi</pre>
After fixing the system, don't forget to umount and export the pools:

<pre>mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID</pre>

Root on ZFS with native encryption

2021-01-07T00:05:15Z

R3: /* Create system datasets */ mount boot

This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool <code>/boot</code>, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit <code>-O keylocation -O keyformat</code> when creating root pool.

= Notes =

UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

= Preparation =

== Setup live environment ==

Download the '''''extended''''' release from https://www.alpinelinux.org/downloads/, as only this version is shipped with ZFS kernel module. Alpine Linux can not load kernel module in live.

Run the following command to setup the live environment, use default <code>none</code> option when asked about disks.

<pre>setup-alpine</pre>
Settings given here will be copied to the target system later by <code>setup-disk</code>.

== Install system utilities ==

Install and setup<code>eudev</code> (a port of systemd <code>udev</code> by gentoo) to get block device names.

<pre>apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
setup-udev</pre>
= Variables =

In this step, we will set some variables to make our installation process easier.

<pre>DISK=/dev/disk/by-id/ata-HXY_120G_YS</pre>
Use unique disk path instead of <code>/dev/sda</code> to ensure the correct partition can be found by ZFS.

Other variables

<pre>TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'</pre>
Create a mountpoint

<pre>MOUNTPOINT=`mktemp -d`</pre>
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.

<pre>poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)</pre>
= Partitioning =

For a single disk, UEFI installation, we need to create at lease 3 partitions: - EFI system partition - Boot pool partition - Root pool partition Since GRUB only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:

<pre>sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool</pre>
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==

<code>Swap</code> support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:

<pre>sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition</pre>
= Create boot and root pool =

As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no <code>feature@</code> is supplied.

Here we explicitly enable those GRUB can support.

<pre>zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2</pre>
Nothing is stored directly under bpool and rpool, hence <code>canmount=off</code>. The respective <code>mountpoint</code> properties are more symbolic than practical.

For root pool all available features are enabled by default

<pre>echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3</pre>
== For multi-disk ==

For mirror:

<pre>zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3</pre>
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Create system datasets =

This layout is intended to separate root file system from persistent files. See https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout for a description.

<pre>zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=legacy -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
mkdir $MOUNTPOINT/boot
mount -t zfs bpool_$poolUUID/BOOT/default $MOUNTPOINT/boot
# ash, default with busybox, does not support array
# this is word splitting
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME</pre>
Depending on your application, separate datasets need to be created for folders inside <code>/var/lib</code>(not itself!)

Here we create several folders for persistent (shared) data, like we just did for <code>/home</code>.

<pre>d='libvirt lxc docker'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done</pre>
<code>lxc</code> is for Linux container, <code>libvirt</code> is for storing virtual machine images, etc.

= Format and mount EFI partition =

Here we use <code>/boot/efi</code> as the mountpoint, which is default for GRUB.

<pre>mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi # need to specify file system</pre>
= System installation =

== Preparation ==

GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=1.

<pre>export ZPOOL_VDEV_NAME_PATH=1</pre>
<code>setup-disk</code> refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.

<pre>sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk</pre>
== setup-disk ==

Run <code>setup-disk</code> to install system to target disk.

<pre>BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT</pre>
Note that grub-probe will still fail despite <code>ZPOOL_VDEV_NAME_PATH=YES</code> variable set above. We will deal with this later inside chroot.

== Chroot ==

<pre>m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh</pre>
=== Finish GRUB installation ===

As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply fix:

<pre>echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile</pre>
Reload

<pre>source /etc/profile</pre>
==== GRUB fails to detect the ZFS filesystem of /boot with BusyBox stat ====

<pre>apk add coreutils</pre>
==== Missing root pool ====

GRUB will fail to detect rpool if rpool has unsupported features, use the following workaround:

<pre>sed -i "s|rpool=.*|rpool=\`zdb -l \${GRUB_DEVICE} \| grep -E '[[:blank:]]name' \| cut -d\\\' -f 2\`|" /etc/grub.d/10_linux</pre>
This replaces GRUB rpool name detection.

==== Generate grub.cfg ====

After applying fixes, finally run

<pre>grub-mkconfig -o /boot/grub/grub.cfg</pre>
=== Importing pools on boot ===

<code>zpool.cache</code> will be added to initramfs and zpool command will import pools contained in this cache.

System will fail to boot without this.

<pre>zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID</pre>
=== Initramfs ===

<code>mkinitfs</code> included in stable Alpine Linux has bugs, before [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 1] and [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76 2] is merged, we need to patch it manually. #### Patch Ensure <code>mkinitfs</code> version is the following

<pre>foolive:/# apk info mkinitfs
mkinitfs-3.4.5-r3 description:</pre>
Then download [https://g.nu8.org/posts/bieaz/setup/alpine/guide/patch/eudev-zfs-mkinitfs-3.4.5.patch|eudev-zfs-mkinitfs-3.4.5.patch], install <code>patch</code> and patch it.

<pre>foolive:~# wget https://g.nu8.org/path-to-patch
foolive:~# apk add patch
foolive:~# cd / # must apply patch at root
foolive:/# patch -Np1 -i /root/eudev-zfs-mkinitfs-3.4.5.patch
patching file etc/mkinitfs/features.d/eudev.files
patching file etc/mkinitfs/features.d/zfs.files
patching file usr/share/mkinitfs/initramfs-init</pre>
==== Add eudev hook and rebuild ====

Add <code>eudev</code> to <code>/etc/mkinitfs/mkinitfs.conf</code>.

<pre>echo 'features="ata base eudev ide scsi usb virtio nvme zfs"' > /etc/mkinitfs/mkinitfs.conf
# order of features is important! this order is tested</pre>
Rebuild initramfs with

<pre>mkinitfs $(ls -1 /lib/modules/)</pre>

=== Mount datasets at boot ===

<pre>rc-update add zfs-mount sysinit</pre>
Mounting <code>/boot</code> dataset with fstab need <code>mountpoint=legacy</code>:

<pre>umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi</pre>
=== Add user ===

<pre>adduser -s /bin/sh -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd</pre>
Root account is accessed via <code>su</code> command with root password.

=== Boot environment manager ===

[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request]. Should be available in edge/test soon.

=== Optional: Enable encrypted swap partition ===

Install <code>cryptsetup</code>

<pre>apk add cryptsetup</pre>
Edit the <code>/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the front of zfs. Add relevant lines in <code>fstab</code> and <code>crypttab</code>. Replace <code>$DISK</code> with actual disk.

<pre>echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab</pre>
Rebuild initramfs with <code>mkinitfs</code>.

= Finish installation =

Take a snapshot for the clean installation for future use and export all pools.

<pre>exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install</pre>
Pools must be exported before reboot, or they will fail to be imported on boot.

<pre>mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID</pre>
= Reboot =

<pre>reboot</pre>
= Recovery in Live environment =

Boot Live environment (extended release) and repeat [[#preparation|Preparation]]

Create a mount point and store encryption password in a variable:

<pre>MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'</pre>
Find the unique UUID of your pool with

<pre>zpool import</pre>
Import rpool without mounting datasets: <code>-N</code> for not mounting all datasets; <code>-R</code> for alternate root.

<pre>poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID</pre>
Load encryption key

<pre>echo $ENCRYPTION_PWD | zfs load-key -a</pre>
As <code>canmount=noauto</code> is set for <code>/</code> dataset, we have to mount it manually. To find the dataset, use

<pre>zfs list rpool_$poolUUID/ROOT</pre>
Mount <code>/</code> dataset

<pre>zfs mount rpool_$UUID/ROOT/$dataset</pre>
Mount other datasets

<pre>zfs mount -a</pre>
Import bpool

<pre>zpool import -N -R $MOUNTPOINT bpool_$UUID</pre>
Find and mount the <code>/boot</code> dataset, same as above.

<pre>zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/$dataset $MOUNTPOINT/boot # legacy mountpoint</pre>
Chroot

<pre>mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh</pre>
After chroot, mount <code>/efi</code>

<pre>mount /boot/efi</pre>
After fixing the system, don't forget to umount and export the pools:

<pre>mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID</pre>

Root on ZFS with native encryption

2021-01-07T00:02:23Z

R3: /* Initramfs */ fix link

This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool <code>/boot</code>, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit <code>-O keylocation -O keyformat</code> when creating root pool.

= Notes =

UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

= Preparation =

== Setup live environment ==

Download the '''''extended''''' release from https://www.alpinelinux.org/downloads/, as only this version is shipped with ZFS kernel module. Alpine Linux can not load kernel module in live.

Run the following command to setup the live environment, use default <code>none</code> option when asked about disks.

<pre>setup-alpine</pre>
Settings given here will be copied to the target system later by <code>setup-disk</code>.

== Install system utilities ==

Install and setup<code>eudev</code> (a port of systemd <code>udev</code> by gentoo) to get block device names.

<pre>apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
setup-udev</pre>
= Variables =

In this step, we will set some variables to make our installation process easier.

<pre>DISK=/dev/disk/by-id/ata-HXY_120G_YS</pre>
Use unique disk path instead of <code>/dev/sda</code> to ensure the correct partition can be found by ZFS.

Other variables

<pre>TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'</pre>
Create a mountpoint

<pre>MOUNTPOINT=`mktemp -d`</pre>
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.

<pre>poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)</pre>
= Partitioning =

For a single disk, UEFI installation, we need to create at lease 3 partitions: - EFI system partition - Boot pool partition - Root pool partition Since GRUB only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:

<pre>sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool</pre>
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==

<code>Swap</code> support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:

<pre>sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition</pre>
= Create boot and root pool =

As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no <code>feature@</code> is supplied.

Here we explicitly enable those GRUB can support.

<pre>zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2</pre>
Nothing is stored directly under bpool and rpool, hence <code>canmount=off</code>. The respective <code>mountpoint</code> properties are more symbolic than practical.

For root pool all available features are enabled by default

<pre>echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3</pre>
== For multi-disk ==

For mirror:

<pre>zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3</pre>
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Create system datasets =

This layout is intended to separate root file system from persistent files. See https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout for a description.

<pre>zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
# ash, default with busybox, does not support array
# this is word splitting
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME</pre>
Depending on your application, separate datasets need to be created for folders inside <code>/var/lib</code>(not itself!)

Here we create several folders for persistent (shared) data, like we just did for <code>/home</code>.

<pre>d='libvirt lxc docker'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done</pre>
<code>lxc</code> is for Linux container, <code>libvirt</code> is for storing virtual machine images, etc.

= Format and mount EFI partition =

Here we use <code>/boot/efi</code> as the mountpoint, which is default for GRUB.

<pre>mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi # need to specify file system</pre>
= System installation =

== Preparation ==

GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=1.

<pre>export ZPOOL_VDEV_NAME_PATH=1</pre>
<code>setup-disk</code> refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.

<pre>sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk</pre>
== setup-disk ==

Run <code>setup-disk</code> to install system to target disk.

<pre>BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT</pre>
Note that grub-probe will still fail despite <code>ZPOOL_VDEV_NAME_PATH=YES</code> variable set above. We will deal with this later inside chroot.

== Chroot ==

<pre>m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh</pre>
=== Finish GRUB installation ===

As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply fix:

<pre>echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile</pre>
Reload

<pre>source /etc/profile</pre>
==== GRUB fails to detect the ZFS filesystem of /boot with BusyBox stat ====

<pre>apk add coreutils</pre>
==== Missing root pool ====

GRUB will fail to detect rpool if rpool has unsupported features, use the following workaround:

<pre>sed -i "s|rpool=.*|rpool=\`zdb -l \${GRUB_DEVICE} \| grep -E '[[:blank:]]name' \| cut -d\\\' -f 2\`|" /etc/grub.d/10_linux</pre>
This replaces GRUB rpool name detection.

==== Generate grub.cfg ====

After applying fixes, finally run

<pre>grub-mkconfig -o /boot/grub/grub.cfg</pre>
=== Importing pools on boot ===

<code>zpool.cache</code> will be added to initramfs and zpool command will import pools contained in this cache.

System will fail to boot without this.

<pre>zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID</pre>
=== Initramfs ===

<code>mkinitfs</code> included in stable Alpine Linux has bugs, before [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 1] and [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76 2] is merged, we need to patch it manually. #### Patch Ensure <code>mkinitfs</code> version is the following

<pre>foolive:/# apk info mkinitfs
mkinitfs-3.4.5-r3 description:</pre>
Then download [https://g.nu8.org/posts/bieaz/setup/alpine/guide/patch/eudev-zfs-mkinitfs-3.4.5.patch|eudev-zfs-mkinitfs-3.4.5.patch], install <code>patch</code> and patch it.

<pre>foolive:~# wget https://g.nu8.org/path-to-patch
foolive:~# apk add patch
foolive:~# cd / # must apply patch at root
foolive:/# patch -Np1 -i /root/eudev-zfs-mkinitfs-3.4.5.patch
patching file etc/mkinitfs/features.d/eudev.files
patching file etc/mkinitfs/features.d/zfs.files
patching file usr/share/mkinitfs/initramfs-init</pre>
==== Add eudev hook and rebuild ====

Add <code>eudev</code> to <code>/etc/mkinitfs/mkinitfs.conf</code>.

<pre>echo 'features="ata base eudev ide scsi usb virtio nvme zfs"' > /etc/mkinitfs/mkinitfs.conf
# order of features is important! this order is tested</pre>
Rebuild initramfs with

<pre>mkinitfs $(ls -1 /lib/modules/)</pre>

=== Mount datasets at boot ===

<pre>rc-update add zfs-mount sysinit</pre>
Mounting <code>/boot</code> dataset with fstab need <code>mountpoint=legacy</code>:

<pre>umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi</pre>
=== Add user ===

<pre>adduser -s /bin/sh -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd</pre>
Root account is accessed via <code>su</code> command with root password.

=== Boot environment manager ===

[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request]. Should be available in edge/test soon.

=== Optional: Enable encrypted swap partition ===

Install <code>cryptsetup</code>

<pre>apk add cryptsetup</pre>
Edit the <code>/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the front of zfs. Add relevant lines in <code>fstab</code> and <code>crypttab</code>. Replace <code>$DISK</code> with actual disk.

<pre>echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab</pre>
Rebuild initramfs with <code>mkinitfs</code>.

= Finish installation =

Take a snapshot for the clean installation for future use and export all pools.

<pre>exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install</pre>
Pools must be exported before reboot, or they will fail to be imported on boot.

<pre>mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID</pre>
= Reboot =

<pre>reboot</pre>
= Recovery in Live environment =

Boot Live environment (extended release) and repeat [[#preparation|Preparation]]

Create a mount point and store encryption password in a variable:

<pre>MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'</pre>
Find the unique UUID of your pool with

<pre>zpool import</pre>
Import rpool without mounting datasets: <code>-N</code> for not mounting all datasets; <code>-R</code> for alternate root.

<pre>poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID</pre>
Load encryption key

<pre>echo $ENCRYPTION_PWD | zfs load-key -a</pre>
As <code>canmount=noauto</code> is set for <code>/</code> dataset, we have to mount it manually. To find the dataset, use

<pre>zfs list rpool_$poolUUID/ROOT</pre>
Mount <code>/</code> dataset

<pre>zfs mount rpool_$UUID/ROOT/$dataset</pre>
Mount other datasets

<pre>zfs mount -a</pre>
Import bpool

<pre>zpool import -N -R $MOUNTPOINT bpool_$UUID</pre>
Find and mount the <code>/boot</code> dataset, same as above.

<pre>zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/$dataset $MOUNTPOINT/boot # legacy mountpoint</pre>
Chroot

<pre>mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh</pre>
After chroot, mount <code>/efi</code>

<pre>mount /boot/efi</pre>
After fixing the system, don't forget to umount and export the pools:

<pre>mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID</pre>

Root on ZFS with native encryption

2021-01-07T00:00:44Z

R3: /* Missing root pool */ fix

This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool <code>/boot</code>, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit <code>-O keylocation -O keyformat</code> when creating root pool.

= Notes =

UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

= Preparation =

== Setup live environment ==

Download the '''''extended''''' release from https://www.alpinelinux.org/downloads/, as only this version is shipped with ZFS kernel module. Alpine Linux can not load kernel module in live.

Run the following command to setup the live environment, use default <code>none</code> option when asked about disks.

<pre>setup-alpine</pre>
Settings given here will be copied to the target system later by <code>setup-disk</code>.

== Install system utilities ==

Install and setup<code>eudev</code> (a port of systemd <code>udev</code> by gentoo) to get block device names.

<pre>apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
setup-udev</pre>
= Variables =

In this step, we will set some variables to make our installation process easier.

<pre>DISK=/dev/disk/by-id/ata-HXY_120G_YS</pre>
Use unique disk path instead of <code>/dev/sda</code> to ensure the correct partition can be found by ZFS.

Other variables

<pre>TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'</pre>
Create a mountpoint

<pre>MOUNTPOINT=`mktemp -d`</pre>
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.

<pre>poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)</pre>
= Partitioning =

For a single disk, UEFI installation, we need to create at lease 3 partitions: - EFI system partition - Boot pool partition - Root pool partition Since GRUB only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:

<pre>sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool</pre>
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==

<code>Swap</code> support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:

<pre>sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition</pre>
= Create boot and root pool =

As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no <code>feature@</code> is supplied.

Here we explicitly enable those GRUB can support.

<pre>zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2</pre>
Nothing is stored directly under bpool and rpool, hence <code>canmount=off</code>. The respective <code>mountpoint</code> properties are more symbolic than practical.

For root pool all available features are enabled by default

<pre>echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3</pre>
== For multi-disk ==

For mirror:

<pre>zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3</pre>
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Create system datasets =

This layout is intended to separate root file system from persistent files. See https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout for a description.

<pre>zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
# ash, default with busybox, does not support array
# this is word splitting
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME</pre>
Depending on your application, separate datasets need to be created for folders inside <code>/var/lib</code>(not itself!)

Here we create several folders for persistent (shared) data, like we just did for <code>/home</code>.

<pre>d='libvirt lxc docker'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done</pre>
<code>lxc</code> is for Linux container, <code>libvirt</code> is for storing virtual machine images, etc.

= Format and mount EFI partition =

Here we use <code>/boot/efi</code> as the mountpoint, which is default for GRUB.

<pre>mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi # need to specify file system</pre>
= System installation =

== Preparation ==

GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=1.

<pre>export ZPOOL_VDEV_NAME_PATH=1</pre>
<code>setup-disk</code> refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.

<pre>sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk</pre>
== setup-disk ==

Run <code>setup-disk</code> to install system to target disk.

<pre>BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT</pre>
Note that grub-probe will still fail despite <code>ZPOOL_VDEV_NAME_PATH=YES</code> variable set above. We will deal with this later inside chroot.

== Chroot ==

<pre>m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh</pre>
=== Finish GRUB installation ===

As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply fix:

<pre>echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile</pre>
Reload

<pre>source /etc/profile</pre>
==== GRUB fails to detect the ZFS filesystem of /boot with BusyBox stat ====

<pre>apk add coreutils</pre>
==== Missing root pool ====

GRUB will fail to detect rpool if rpool has unsupported features, use the following workaround:

<pre>sed -i "s|rpool=.*|rpool=\`zdb -l \${GRUB_DEVICE} \| grep -E '[[:blank:]]name' \| cut -d\\\' -f 2\`|" /etc/grub.d/10_linux</pre>
This replaces GRUB rpool name detection.

==== Generate grub.cfg ====

After applying fixes, finally run

<pre>grub-mkconfig -o /boot/grub/grub.cfg</pre>
=== Importing pools on boot ===

<code>zpool.cache</code> will be added to initramfs and zpool command will import pools contained in this cache.

System will fail to boot without this.

<pre>zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID</pre>
=== Initramfs ===

<code>mkinitfs</code> included in stable Alpine Linux has bugs, before [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 1] and [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76 2] is merged, we need to patch it manually. #### Patch Ensure <code>mkinitfs</code> version is the following

<pre>foolive:/# apk info mkinitfs
mkinitfs-3.4.5-r3 description:</pre>
Then download [[patch/eudev-zfs-mkinitfs-3.4.5.patch|eudev-zfs-mkinitfs-3.4.5.patch]], install <code>patch</code> and patch it.

<pre>foolive:~# wget https://g.nu8.org/path-to-patch
foolive:~# apk add patch
foolive:~# cd / # must apply patch at root
foolive:/# patch -Np1 -i /root/eudev-zfs-mkinitfs-3.4.5.patch
patching file etc/mkinitfs/features.d/eudev.files
patching file etc/mkinitfs/features.d/zfs.files
patching file usr/share/mkinitfs/initramfs-init</pre>
==== Add eudev hook and rebuild ====

Add <code>eudev</code> to <code>/etc/mkinitfs/mkinitfs.conf</code>.

<pre>echo 'features="ata base eudev ide scsi usb virtio nvme zfs"' > /etc/mkinitfs/mkinitfs.conf
# order of features is important! this order is tested</pre>
Rebuild initramfs with

<pre>mkinitfs $(ls -1 /lib/modules/)</pre>
=== Mount datasets at boot ===

<pre>rc-update add zfs-mount sysinit</pre>
Mounting <code>/boot</code> dataset with fstab need <code>mountpoint=legacy</code>:

<pre>umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi</pre>
=== Add user ===

<pre>adduser -s /bin/sh -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd</pre>
Root account is accessed via <code>su</code> command with root password.

=== Boot environment manager ===

[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request]. Should be available in edge/test soon.

=== Optional: Enable encrypted swap partition ===

Install <code>cryptsetup</code>

<pre>apk add cryptsetup</pre>
Edit the <code>/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the front of zfs. Add relevant lines in <code>fstab</code> and <code>crypttab</code>. Replace <code>$DISK</code> with actual disk.

<pre>echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab</pre>
Rebuild initramfs with <code>mkinitfs</code>.

= Finish installation =

Take a snapshot for the clean installation for future use and export all pools.

<pre>exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install</pre>
Pools must be exported before reboot, or they will fail to be imported on boot.

<pre>mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID</pre>
= Reboot =

<pre>reboot</pre>
= Recovery in Live environment =

Boot Live environment (extended release) and repeat [[#preparation|Preparation]]

Create a mount point and store encryption password in a variable:

<pre>MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'</pre>
Find the unique UUID of your pool with

<pre>zpool import</pre>
Import rpool without mounting datasets: <code>-N</code> for not mounting all datasets; <code>-R</code> for alternate root.

<pre>poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID</pre>
Load encryption key

<pre>echo $ENCRYPTION_PWD | zfs load-key -a</pre>
As <code>canmount=noauto</code> is set for <code>/</code> dataset, we have to mount it manually. To find the dataset, use

<pre>zfs list rpool_$poolUUID/ROOT</pre>
Mount <code>/</code> dataset

<pre>zfs mount rpool_$UUID/ROOT/$dataset</pre>
Mount other datasets

<pre>zfs mount -a</pre>
Import bpool

<pre>zpool import -N -R $MOUNTPOINT bpool_$UUID</pre>
Find and mount the <code>/boot</code> dataset, same as above.

<pre>zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/$dataset $MOUNTPOINT/boot # legacy mountpoint</pre>
Chroot

<pre>mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh</pre>
After chroot, mount <code>/efi</code>

<pre>mount /boot/efi</pre>
After fixing the system, don't forget to umount and export the pools:

<pre>mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID</pre>

Root on ZFS with native encryption

2021-01-06T23:59:38Z

R3: update

This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool <code>/boot</code>, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit <code>-O keylocation -O keyformat</code> when creating root pool.

= Notes =

UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

= Preparation =

== Setup live environment ==

Download the '''''extended''''' release from https://www.alpinelinux.org/downloads/, as only this version is shipped with ZFS kernel module. Alpine Linux can not load kernel module in live.

Run the following command to setup the live environment, use default <code>none</code> option when asked about disks.

<pre>setup-alpine</pre>
Settings given here will be copied to the target system later by <code>setup-disk</code>.

== Install system utilities ==

Install and setup<code>eudev</code> (a port of systemd <code>udev</code> by gentoo) to get block device names.

<pre>apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
setup-udev</pre>
= Variables =

In this step, we will set some variables to make our installation process easier.

<pre>DISK=/dev/disk/by-id/ata-HXY_120G_YS</pre>
Use unique disk path instead of <code>/dev/sda</code> to ensure the correct partition can be found by ZFS.

Other variables

<pre>TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'</pre>
Create a mountpoint

<pre>MOUNTPOINT=`mktemp -d`</pre>
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.

<pre>poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)</pre>
= Partitioning =

For a single disk, UEFI installation, we need to create at lease 3 partitions: - EFI system partition - Boot pool partition - Root pool partition Since GRUB only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:

<pre>sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool</pre>
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==

<code>Swap</code> support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:

<pre>sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition</pre>
= Create boot and root pool =

As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no <code>feature@</code> is supplied.

Here we explicitly enable those GRUB can support.

<pre>zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2</pre>
Nothing is stored directly under bpool and rpool, hence <code>canmount=off</code>. The respective <code>mountpoint</code> properties are more symbolic than practical.

For root pool all available features are enabled by default

<pre>echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3</pre>
== For multi-disk ==

For mirror:

<pre>zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3</pre>
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Create system datasets =

This layout is intended to separate root file system from persistent files. See https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout for a description.

<pre>zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
# ash, default with busybox, does not support array
# this is word splitting
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME</pre>
Depending on your application, separate datasets need to be created for folders inside <code>/var/lib</code>(not itself!)

Here we create several folders for persistent (shared) data, like we just did for <code>/home</code>.

<pre>d='libvirt lxc docker'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done</pre>
<code>lxc</code> is for Linux container, <code>libvirt</code> is for storing virtual machine images, etc.

= Format and mount EFI partition =

Here we use <code>/boot/efi</code> as the mountpoint, which is default for GRUB.

<pre>mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi # need to specify file system</pre>
= System installation =

== Preparation ==

GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=1.

<pre>export ZPOOL_VDEV_NAME_PATH=1</pre>
<code>setup-disk</code> refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.

<pre>sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk</pre>
== setup-disk ==

Run <code>setup-disk</code> to install system to target disk.

<pre>BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT</pre>
Note that grub-probe will still fail despite <code>ZPOOL_VDEV_NAME_PATH=YES</code> variable set above. We will deal with this later inside chroot.

== Chroot ==

<pre>m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh</pre>
=== Finish GRUB installation ===

As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply fix:

<pre>echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile</pre>
Reload

<pre>source /etc/profile</pre>
==== GRUB fails to detect the ZFS filesystem of /boot with BusyBox stat ====

<pre>apk add coreutils</pre>
==== Missing root pool ====

Before [https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html this patch] is merged, use the following workaround:

<pre>sed -i "s|rpool=.*|rpool=\`zdb -l \${GRUB_DEVICE} \| grep -E '[[:blank:]]name' \| cut -d\\\' -f 2\`|" /etc/grub.d/10_linux</pre>
This replaces GRUB rpool name detection.

==== Generate grub.cfg ====

After applying fixes, finally run

<pre>grub-mkconfig -o /boot/grub/grub.cfg</pre>
=== Importing pools on boot ===

<code>zpool.cache</code> will be added to initramfs and zpool command will import pools contained in this cache.

System will fail to boot without this.

<pre>zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID</pre>
=== Initramfs ===

<code>mkinitfs</code> included in stable Alpine Linux has bugs, before [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 1] and [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76 2] is merged, we need to patch it manually. #### Patch Ensure <code>mkinitfs</code> version is the following

<pre>foolive:/# apk info mkinitfs
mkinitfs-3.4.5-r3 description:</pre>
Then download [[patch/eudev-zfs-mkinitfs-3.4.5.patch|eudev-zfs-mkinitfs-3.4.5.patch]], install <code>patch</code> and patch it.

<pre>foolive:~# wget https://g.nu8.org/path-to-patch
foolive:~# apk add patch
foolive:~# cd / # must apply patch at root
foolive:/# patch -Np1 -i /root/eudev-zfs-mkinitfs-3.4.5.patch
patching file etc/mkinitfs/features.d/eudev.files
patching file etc/mkinitfs/features.d/zfs.files
patching file usr/share/mkinitfs/initramfs-init</pre>
==== Add eudev hook and rebuild ====

Add <code>eudev</code> to <code>/etc/mkinitfs/mkinitfs.conf</code>.

<pre>echo 'features="ata base eudev ide scsi usb virtio nvme zfs"' > /etc/mkinitfs/mkinitfs.conf
# order of features is important! this order is tested</pre>
Rebuild initramfs with

<pre>mkinitfs $(ls -1 /lib/modules/)</pre>
=== Mount datasets at boot ===

<pre>rc-update add zfs-mount sysinit</pre>
Mounting <code>/boot</code> dataset with fstab need <code>mountpoint=legacy</code>:

<pre>umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi</pre>
=== Add user ===

<pre>adduser -s /bin/sh -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd</pre>
Root account is accessed via <code>su</code> command with root password.

=== Boot environment manager ===

[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request]. Should be available in edge/test soon.

=== Optional: Enable encrypted swap partition ===

Install <code>cryptsetup</code>

<pre>apk add cryptsetup</pre>
Edit the <code>/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the front of zfs. Add relevant lines in <code>fstab</code> and <code>crypttab</code>. Replace <code>$DISK</code> with actual disk.

<pre>echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab</pre>
Rebuild initramfs with <code>mkinitfs</code>.

= Finish installation =

Take a snapshot for the clean installation for future use and export all pools.

<pre>exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install</pre>
Pools must be exported before reboot, or they will fail to be imported on boot.

<pre>mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID</pre>
= Reboot =

<pre>reboot</pre>
= Recovery in Live environment =

Boot Live environment (extended release) and repeat [[#preparation|Preparation]]

Create a mount point and store encryption password in a variable:

<pre>MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'</pre>
Find the unique UUID of your pool with

<pre>zpool import</pre>
Import rpool without mounting datasets: <code>-N</code> for not mounting all datasets; <code>-R</code> for alternate root.

<pre>poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID</pre>
Load encryption key

<pre>echo $ENCRYPTION_PWD | zfs load-key -a</pre>
As <code>canmount=noauto</code> is set for <code>/</code> dataset, we have to mount it manually. To find the dataset, use

<pre>zfs list rpool_$poolUUID/ROOT</pre>
Mount <code>/</code> dataset

<pre>zfs mount rpool_$UUID/ROOT/$dataset</pre>
Mount other datasets

<pre>zfs mount -a</pre>
Import bpool

<pre>zpool import -N -R $MOUNTPOINT bpool_$UUID</pre>
Find and mount the <code>/boot</code> dataset, same as above.

<pre>zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/$dataset $MOUNTPOINT/boot # legacy mountpoint</pre>
Chroot

<pre>mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh</pre>
After chroot, mount <code>/efi</code>

<pre>mount /boot/efi</pre>
After fixing the system, don't forget to umount and export the pools:

<pre>mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID</pre>

Root on ZFS with native encryption

2021-01-06T23:49:20Z

R3: /* Missing root pool */ update

= Useful links =
*[https://openzfs.github.io/openzfs-docs/Getting%20Started/ OpenZFS Getting Started]
*[https://g.nu8.org/posts/bieaz/setup/alpine/guide/ Encrypted ZFS with boot environment support]

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. BusyBox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

== DO NOT set bootfs property! ==
Do not set {{ic|bootfs}} on any pool!

It will override {{ic|1=root=ZFS=rpool/ROOT/dataset}} kernel parameter and render boot environment menu in GRUB '''INVALID'''.

As GRUB support of ZFS is read-only, you will need to boot into live environment to unset this property if `bootfs` dataset is broken.

Boot environment menu is currently only available for GRUB. More info see [https://gitlab.com/m_zhou/bieaz bieaz boot environment manager readme].

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select default option {{ic|1=disk=none}} at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}
Depending on your application, separate datasets need to be created for folders inside {{ic|/var/lib}}(not itself!)

Here we create several folders for persistent (shared) data, like we just did for {{ic|/home}}.
d='libvirt lxc docker'
for i in d; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done
{{ic|lxc}} is for Linux container, {{ic|libvirt}} is for storing virtual machine images, etc.

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi
For multi-disk setup, a cron job needs to be configured to sync contents. It should be similar to [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Multi-ESP this article].

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in [[#GRUB fixes]].
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from BusyBox.
See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from BusyBox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
sed -i "s|rpool=.*|rpool=\`zdb -l \${GRUB_DEVICE} \| grep -E '[[:blank:]]name' \| cut -d\\\' -f 2\`|" /etc/grub.d/10_linux

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.

System will fail to boot without this.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76/diffs this merge request].
== Enable persistent device names ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77/diffs this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd
Root account is accessed via {{ic|su}} command with root password.

Optionally install {{ic|sudo}} to disable root password and use user's own password instead.

= Boot environment manager =
[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request]. Should be available in edge/test soon.

= Optional: Desktop Environment =
See [[#Wayland-based_lightweight_desktop]].

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
== Barebone ==
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K
== Wayland-based lightweight desktop ==
This setup is based on Sway Window Manager and Qt apps.

Encrypted swap
apk add cryptsetup
Sway Window Manager and basic utilities
apk add sway swayidle swaylock grim i3status
Terminal
apk add alacritty
Sound
apk add alsa-utils
Utilities
apk add vim mutt isync lynx git p7zip proxychains-ng
Qt-based desktop environment, with dark theme, fdo keyring, file manager and PDF viewer
apk add qt5-qtwayland kvantum keepassxc pcmanfm zathura-pdf-poppler
Play videos with hardware accelerated decoding
apk add mpv youtube-dl libva-intel-driver
Firefox
apk add firefox-esr
Add MTP (connect to Android phones) and samba support to file manager
apk add gvfs-smb gvfs-mtp
Add dark GTK theme (Adwaita-dark), HiDPI mouse cursor for Sway, GTK icons
apk add gnome-themes-extra
Stat
*rpool used 1.11G
*bpool used 26.6M

= Recovery in Live environment =
Boot Live environment (extended release) and install packages:
setup-alpine # basic settings: keyboard layout, timezone ...
apk-add zfs eudev # zfs-utils and persistent device name support
setup-udev # populate persistent names
modprobe zfs # load kernel module
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-06T23:46:10Z

R3: links

= Useful links =
*[https://openzfs.github.io/openzfs-docs/Getting%20Started/ OpenZFS Getting Started]
*[https://g.nu8.org/posts/bieaz/setup/alpine/guide/ Encrypted ZFS with boot environment support]

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. BusyBox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

== DO NOT set bootfs property! ==
Do not set {{ic|bootfs}} on any pool!

It will override {{ic|1=root=ZFS=rpool/ROOT/dataset}} kernel parameter and render boot environment menu in GRUB '''INVALID'''.

As GRUB support of ZFS is read-only, you will need to boot into live environment to unset this property if `bootfs` dataset is broken.

Boot environment menu is currently only available for GRUB. More info see [https://gitlab.com/m_zhou/bieaz bieaz boot environment manager readme].

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select default option {{ic|1=disk=none}} at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}
Depending on your application, separate datasets need to be created for folders inside {{ic|/var/lib}}(not itself!)

Here we create several folders for persistent (shared) data, like we just did for {{ic|/home}}.
d='libvirt lxc docker'
for i in d; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done
{{ic|lxc}} is for Linux container, {{ic|libvirt}} is for storing virtual machine images, etc.

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi
For multi-disk setup, a cron job needs to be configured to sync contents. It should be similar to [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Multi-ESP this article].

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in [[#GRUB fixes]].
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from BusyBox.
See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from BusyBox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
with
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
And you must install
apk add util-linux
since {{ic|blkid}} from BusyBox does not support ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.

System will fail to boot without this.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76/diffs this merge request].
== Enable persistent device names ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77/diffs this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd
Root account is accessed via {{ic|su}} command with root password.

Optionally install {{ic|sudo}} to disable root password and use user's own password instead.

= Boot environment manager =
[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request]. Should be available in edge/test soon.

= Optional: Desktop Environment =
See [[#Wayland-based_lightweight_desktop]].

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
== Barebone ==
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K
== Wayland-based lightweight desktop ==
This setup is based on Sway Window Manager and Qt apps.

Encrypted swap
apk add cryptsetup
Sway Window Manager and basic utilities
apk add sway swayidle swaylock grim i3status
Terminal
apk add alacritty
Sound
apk add alsa-utils
Utilities
apk add vim mutt isync lynx git p7zip proxychains-ng
Qt-based desktop environment, with dark theme, fdo keyring, file manager and PDF viewer
apk add qt5-qtwayland kvantum keepassxc pcmanfm zathura-pdf-poppler
Play videos with hardware accelerated decoding
apk add mpv youtube-dl libva-intel-driver
Firefox
apk add firefox-esr
Add MTP (connect to Android phones) and samba support to file manager
apk add gvfs-smb gvfs-mtp
Add dark GTK theme (Adwaita-dark), HiDPI mouse cursor for Sway, GTK icons
apk add gnome-themes-extra
Stat
*rpool used 1.11G
*bpool used 26.6M

= Recovery in Live environment =
Boot Live environment (extended release) and install packages:
setup-alpine # basic settings: keyboard layout, timezone ...
apk-add zfs eudev # zfs-utils and persistent device name support
setup-udev # populate persistent names
modprobe zfs # load kernel module
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-06T23:35:43Z

R3: Undo revision 18485 by R3 (talk)

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. BusyBox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

== DO NOT set bootfs property! ==
Do not set {{ic|bootfs}} on any pool!

It will override {{ic|1=root=ZFS=rpool/ROOT/dataset}} kernel parameter and render boot environment menu in GRUB '''INVALID'''.

As GRUB support of ZFS is read-only, you will need to boot into live environment to unset this property if `bootfs` dataset is broken.

Boot environment menu is currently only available for GRUB. More info see [https://gitlab.com/m_zhou/bieaz bieaz boot environment manager readme].

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select default option {{ic|1=disk=none}} at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}
Depending on your application, separate datasets need to be created for folders inside {{ic|/var/lib}}(not itself!)

Here we create several folders for persistent (shared) data, like we just did for {{ic|/home}}.
d='libvirt lxc docker'
for i in d; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done
{{ic|lxc}} is for Linux container, {{ic|libvirt}} is for storing virtual machine images, etc.

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi
For multi-disk setup, a cron job needs to be configured to sync contents. It should be similar to [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Multi-ESP this article].

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in [[#GRUB fixes]].
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from BusyBox.
See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from BusyBox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
with
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
And you must install
apk add util-linux
since {{ic|blkid}} from BusyBox does not support ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.

System will fail to boot without this.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76/diffs this merge request].
== Enable persistent device names ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77/diffs this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd
Root account is accessed via {{ic|su}} command with root password.

Optionally install {{ic|sudo}} to disable root password and use user's own password instead.

= Boot environment manager =
[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request]. Should be available in edge/test soon.

= Optional: Desktop Environment =
See [[#Wayland-based_lightweight_desktop]].

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
== Barebone ==
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K
== Wayland-based lightweight desktop ==
This setup is based on Sway Window Manager and Qt apps.

Encrypted swap
apk add cryptsetup
Sway Window Manager and basic utilities
apk add sway swayidle swaylock grim i3status
Terminal
apk add alacritty
Sound
apk add alsa-utils
Utilities
apk add vim mutt isync lynx git p7zip proxychains-ng
Qt-based desktop environment, with dark theme, fdo keyring, file manager and PDF viewer
apk add qt5-qtwayland kvantum keepassxc pcmanfm zathura-pdf-poppler
Play videos with hardware accelerated decoding
apk add mpv youtube-dl libva-intel-driver
Firefox
apk add firefox-esr
Add MTP (connect to Android phones) and samba support to file manager
apk add gvfs-smb gvfs-mtp
Add dark GTK theme (Adwaita-dark), HiDPI mouse cursor for Sway, GTK icons
apk add gnome-themes-extra
Stat
*rpool used 1.11G
*bpool used 26.6M

= Recovery in Live environment =
Boot Live environment (extended release) and install packages:
setup-alpine # basic settings: keyboard layout, timezone ...
apk-add zfs eudev # zfs-utils and persistent device name support
setup-udev # populate persistent names
modprobe zfs # load kernel module
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-05T07:00:17Z

R3: add links instead

= Setting up Alpine Linux using ZFS with a pool that uses ZFS' native encryption capabilities =

== Useful links ==
*[https://openzfs.github.io/openzfs-docs/Getting%20Started/ OpenZFS Getting Started]
*[https://g.nu8.org/posts/bieaz/setup/alpine/guide/ Encrypted ZFS with boot environment support]

== Download ==

Download the '''extended''' release from https://www.alpinelinux.org/downloads/ as only it contains the zfs kernel mods at the time of this writing (2020.07.10)

Write it to a USB and boot from it.

== Initial setup ==

Run the following

setup-alpine

Answer all the questions, and hit ctrl-c when promted for what disk you'd like to use.

== OPTIONAL ==

This section is optional and it assumes internet connectivity. You may enable sshd so you can ssh into the box and copy and paste the rest of the commands into my terminal window from these instructions.

Edit `/etc/ssh/sshd_config` and search for `Permit`. Change the value after `PermitRootLogin` to read `yes`

save and exit to shell. Run `service sshd restart`

Now you can ssh in as root. Do not forget to go back and comment this line out when you're done since it will be enabled on your resulting machine. You will be reminded again at the end of this doc.

== Add needed packages ==

apk add zfs sfdisk e2fsprogs syslinux

== Create our partitions ==

We're assuming `/dev/sda` here and in the rest of the document but you can use whatever you need to. To see a list, type: `sfdisk -l`

echo -e "/dev/sda1: start=1M,size=100M,bootable\n/dev/sda2: start=101M" | sfdisk --quiet --label dos /dev/sda

== Create device nodes ==

mdev -s

== Create the /boot filesystem ==

mkfs.ext4 /dev/sda1

== Create the root filesystem using zfs ==

modprobe zfs
zpool create -f -o ashift=12 \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \
-O encryption=aes-256-gcm -O keylocation=prompt -O keyformat=passphrase \
-O mountpoint=/ -R /mnt \
rpool /dev/sda2

You will have to enter your passphrase at this point. Choose wisely, as your passphrase is most likely [https://gitlab.com/cryptsetup/cryptsetup/wikis/FrequentlyAskedQuestions#5-security-aspects the weakest link in this setup].

A few notes on the options supplied to zpool:

- `ashift=12` is recommended here because many drives today have 4KiB (or larger) physical sectors, even though they present 512B logical sectors

- `acltype=posixacl` enables POSIX ACLs globally

- `normalization=formD` eliminates some corner cases relating to UTF-8 filename normalization. It also enables `utf8only=on`, meaning that only files with valid UTF-8 filenames will be accepted.

- `xattr=sa` vastly improves the performance of extended attributes, but is Linux-only. If you care about using this pool on other OpenZFS implementation don't specify this option.

After completing this, confirm that the pool has been created:

# zpool status

Should return something like:

pool: rpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
sda2 ONLINE 0 0 0

errors: No known data errors

== Create the required datasets and mount root ==

zfs create -o mountpoint=none -o canmount=off rpool/ROOT
zfs create -o mountpoint=legacy rpool/ROOT/alpine
mount -t zfs rpool/ROOT/alpine /mnt/

== Mount the `/boot` filesystem ==

mkdir /mnt/boot/
mount -t ext4 /dev/sda1 /mnt/boot/

=== Enable ZFS' services ===

rc-update add zfs-import sysinit
rc-update add zfs-mount sysinit

== Install Alpine Linux ==

setup-disk /mnt
dd if=/usr/share/syslinux/mbr.bin of=/dev/sda # write mbr so we can boot

== Reboot and enjoy! ==

😉

'''NOTE:'''
If you went with the optional step, be sure to disable root login after you reboot.

Root on ZFS with native encryption

2021-01-04T10:13:47Z

R3: /* Importing pools on boot */ warning

User:R3/Root on ZFS with Native Encryption

2021-01-04T09:43:10Z

R3: personal page

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. BusyBox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

== DO NOT set bootfs property! ==
Do not set {{ic|bootfs}} on any pool!

It will override {{ic|1=root=ZFS=rpool/ROOT/dataset}} kernel parameter and render boot environment menu in GRUB '''INVALID'''.

As GRUB support of ZFS is read-only, you will need to boot into live environment to unset this property if `bootfs` dataset is broken.

Boot environment menu is currently only available for GRUB. More info see [https://gitlab.com/m_zhou/bieaz bieaz boot environment manager readme].

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select default option {{ic|1=disk=none}} at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}
Depending on your application, separate datasets need to be created for folders inside {{ic|/var/lib}}(not itself!)

Here we create several folders for persistent (shared) data, like we just did for {{ic|/home}}.
d='libvirt lxc docker'
for i in d; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done
{{ic|lxc}} is for Linux container, {{ic|libvirt}} is for storing virtual machine images, etc.

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi
For multi-disk setup, a cron job needs to be configured to sync contents. It should be similar to [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Multi-ESP this article].

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in [[#GRUB fixes]].
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from BusyBox.
See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from BusyBox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
with
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
And you must install
apk add util-linux
since {{ic|blkid}} from BusyBox does not support ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76/diffs this merge request].
== Enable persistent device names ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77/diffs this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd
Root account is accessed via {{ic|su}} command with root password.

Optionally install {{ic|sudo}} to disable root password and use user's own password instead.

= Boot environment manager =
[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request]. Should be available in edge/test soon.

= Optional: Desktop Environment =
See [[#Wayland-based_lightweight_desktop]].

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
== Barebone ==
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K
== Wayland-based lightweight desktop ==
This setup is based on Sway Window Manager and Qt apps.

Encrypted swap
apk add cryptsetup
Sway Window Manager and basic utilities
apk add sway swayidle swaylock grim i3status
Terminal
apk add alacritty
Sound
apk add alsa-utils
Utilities
apk add vim mutt isync lynx git p7zip proxychains-ng
Qt-based desktop environment, with dark theme, fdo keyring, file manager and PDF viewer
apk add qt5-qtwayland kvantum keepassxc pcmanfm zathura-pdf-poppler
Play videos with hardware accelerated decoding
apk add mpv youtube-dl libva-intel-driver
Firefox
apk add firefox-esr
Add MTP (connect to Android phones) and samba support to file manager
apk add gvfs-smb gvfs-mtp
Add dark GTK theme (Adwaita-dark), HiDPI mouse cursor for Sway, GTK icons
apk add gnome-themes-extra
Stat
*rpool used 1.11G
*bpool used 26.6M

= Recovery in Live environment =
Boot Live environment (extended release) and install packages:
setup-alpine # basic settings: keyboard layout, timezone ...
apk-add zfs eudev # zfs-utils and persistent device name support
setup-udev # populate persistent names
modprobe zfs # load kernel module
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-04T09:42:30Z

R3: /* Boot environment manager */ note

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. BusyBox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

== DO NOT set bootfs property! ==
Do not set {{ic|bootfs}} on any pool!

It will override {{ic|1=root=ZFS=rpool/ROOT/dataset}} kernel parameter and render boot environment menu in GRUB '''INVALID'''.

As GRUB support of ZFS is read-only, you will need to boot into live environment to unset this property if `bootfs` dataset is broken.

Boot environment menu is currently only available for GRUB. More info see [https://gitlab.com/m_zhou/bieaz bieaz boot environment manager readme].

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select default option {{ic|1=disk=none}} at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}
Depending on your application, separate datasets need to be created for folders inside {{ic|/var/lib}}(not itself!)

Here we create several folders for persistent (shared) data, like we just did for {{ic|/home}}.
d='libvirt lxc docker'
for i in d; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done
{{ic|lxc}} is for Linux container, {{ic|libvirt}} is for storing virtual machine images, etc.

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi
For multi-disk setup, a cron job needs to be configured to sync contents. It should be similar to [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Multi-ESP this article].

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in [[#GRUB fixes]].
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from BusyBox.
See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from BusyBox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
with
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
And you must install
apk add util-linux
since {{ic|blkid}} from BusyBox does not support ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76/diffs this merge request].
== Enable persistent device names ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77/diffs this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd
Root account is accessed via {{ic|su}} command with root password.

Optionally install {{ic|sudo}} to disable root password and use user's own password instead.

= Boot environment manager =
[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request]. Should be available in edge/test soon.

= Optional: Desktop Environment =
See [[#Wayland-based_lightweight_desktop]].

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
== Barebone ==
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K
== Wayland-based lightweight desktop ==
This setup is based on Sway Window Manager and Qt apps.

Encrypted swap
apk add cryptsetup
Sway Window Manager and basic utilities
apk add sway swayidle swaylock grim i3status
Terminal
apk add alacritty
Sound
apk add alsa-utils
Utilities
apk add vim mutt isync lynx git p7zip proxychains-ng
Qt-based desktop environment, with dark theme, fdo keyring, file manager and PDF viewer
apk add qt5-qtwayland kvantum keepassxc pcmanfm zathura-pdf-poppler
Play videos with hardware accelerated decoding
apk add mpv youtube-dl libva-intel-driver
Firefox
apk add firefox-esr
Add MTP (connect to Android phones) and samba support to file manager
apk add gvfs-smb gvfs-mtp
Add dark GTK theme (Adwaita-dark), HiDPI mouse cursor for Sway, GTK icons
apk add gnome-themes-extra
Stat
*rpool used 1.11G
*bpool used 26.6M

= Recovery in Live environment =
Boot Live environment (extended release) and install packages:
setup-alpine # basic settings: keyboard layout, timezone ...
apk-add zfs eudev # zfs-utils and persistent device name support
setup-udev # populate persistent names
modprobe zfs # load kernel module
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-04T09:42:02Z

R3: fix link

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. BusyBox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

== DO NOT set bootfs property! ==
Do not set {{ic|bootfs}} on any pool!

It will override {{ic|1=root=ZFS=rpool/ROOT/dataset}} kernel parameter and render boot environment menu in GRUB '''INVALID'''.

As GRUB support of ZFS is read-only, you will need to boot into live environment to unset this property if `bootfs` dataset is broken.

Boot environment menu is currently only available for GRUB. More info see [https://gitlab.com/m_zhou/bieaz bieaz boot environment manager readme].

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select default option {{ic|1=disk=none}} at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}
Depending on your application, separate datasets need to be created for folders inside {{ic|/var/lib}}(not itself!)

Here we create several folders for persistent (shared) data, like we just did for {{ic|/home}}.
d='libvirt lxc docker'
for i in d; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done
{{ic|lxc}} is for Linux container, {{ic|libvirt}} is for storing virtual machine images, etc.

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi
For multi-disk setup, a cron job needs to be configured to sync contents. It should be similar to [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Multi-ESP this article].

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in [[#GRUB fixes]].
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from BusyBox.
See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from BusyBox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
with
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
And you must install
apk add util-linux
since {{ic|blkid}} from BusyBox does not support ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76/diffs this merge request].
== Enable persistent device names ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77/diffs this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd
Root account is accessed via {{ic|su}} command with root password.

Optionally install {{ic|sudo}} to disable root password and use user's own password instead.

= Boot environment manager =
[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request].

= Optional: Desktop Environment =
See [[#Wayland-based_lightweight_desktop]].

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
== Barebone ==
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K
== Wayland-based lightweight desktop ==
This setup is based on Sway Window Manager and Qt apps.

Encrypted swap
apk add cryptsetup
Sway Window Manager and basic utilities
apk add sway swayidle swaylock grim i3status
Terminal
apk add alacritty
Sound
apk add alsa-utils
Utilities
apk add vim mutt isync lynx git p7zip proxychains-ng
Qt-based desktop environment, with dark theme, fdo keyring, file manager and PDF viewer
apk add qt5-qtwayland kvantum keepassxc pcmanfm zathura-pdf-poppler
Play videos with hardware accelerated decoding
apk add mpv youtube-dl libva-intel-driver
Firefox
apk add firefox-esr
Add MTP (connect to Android phones) and samba support to file manager
apk add gvfs-smb gvfs-mtp
Add dark GTK theme (Adwaita-dark), HiDPI mouse cursor for Sway, GTK icons
apk add gnome-themes-extra
Stat
*rpool used 1.11G
*bpool used 26.6M

= Recovery in Live environment =
Boot Live environment (extended release) and install packages:
setup-alpine # basic settings: keyboard layout, timezone ...
apk-add zfs eudev # zfs-utils and persistent device name support
setup-udev # populate persistent names
modprobe zfs # load kernel module
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-04T09:41:25Z

R3: /* Boot environment manager */ de

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. BusyBox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

== DO NOT set bootfs property! ==
Do not set {{ic|bootfs}} on any pool!

It will override {{ic|1=root=ZFS=rpool/ROOT/dataset}} kernel parameter and render boot environment menu in GRUB '''INVALID'''.

As GRUB support of ZFS is read-only, you will need to boot into live environment to unset this property if `bootfs` dataset is broken.

Boot environment menu is currently only available for GRUB. More info see [https://gitlab.com/m_zhou/bieaz bieaz boot environment manager readme].

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select default option {{ic|1=disk=none}} at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}
Depending on your application, separate datasets need to be created for folders inside {{ic|/var/lib}}(not itself!)

Here we create several folders for persistent (shared) data, like we just did for {{ic|/home}}.
d='libvirt lxc docker'
for i in d; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done
{{ic|lxc}} is for Linux container, {{ic|libvirt}} is for storing virtual machine images, etc.

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi
For multi-disk setup, a cron job needs to be configured to sync contents. It should be similar to [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Multi-ESP this article].

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in [#GRUB fixes].
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from BusyBox.
See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from BusyBox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
with
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
And you must install
apk add util-linux
since {{ic|blkid}} from BusyBox does not support ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76/diffs this merge request].
== Enable persistent device names ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77/diffs this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd
Root account is accessed via {{ic|su}} command with root password.

Optionally install {{ic|sudo}} to disable root password and use user's own password instead.

= Boot environment manager =
[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request].

= Optional: Desktop Environment =
See [#Wayland-based_lightweight_desktop].

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
== Barebone ==
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K
== Wayland-based lightweight desktop ==
This setup is based on Sway Window Manager and Qt apps.

Encrypted swap
apk add cryptsetup
Sway Window Manager and basic utilities
apk add sway swayidle swaylock grim i3status
Terminal
apk add alacritty
Sound
apk add alsa-utils
Utilities
apk add vim mutt isync lynx git p7zip proxychains-ng
Qt-based desktop environment, with dark theme, fdo keyring, file manager and PDF viewer
apk add qt5-qtwayland kvantum keepassxc pcmanfm zathura-pdf-poppler
Play videos with hardware accelerated decoding
apk add mpv youtube-dl libva-intel-driver
Firefox
apk add firefox-esr
Add MTP (connect to Android phones) and samba support to file manager
apk add gvfs-smb gvfs-mtp
Add dark GTK theme (Adwaita-dark), HiDPI mouse cursor for Sway, GTK icons
apk add gnome-themes-extra
Stat
*rpool used 1.11G
*bpool used 26.6M

= Recovery in Live environment =
Boot Live environment (extended release) and install packages:
setup-alpine # basic settings: keyboard layout, timezone ...
apk-add zfs eudev # zfs-utils and persistent device name support
setup-udev # populate persistent names
modprobe zfs # load kernel module
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-04T09:39:26Z

R3: /* Disk space stat */ sway

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. BusyBox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

== DO NOT set bootfs property! ==
Do not set {{ic|bootfs}} on any pool!

It will override {{ic|1=root=ZFS=rpool/ROOT/dataset}} kernel parameter and render boot environment menu in GRUB '''INVALID'''.

As GRUB support of ZFS is read-only, you will need to boot into live environment to unset this property if `bootfs` dataset is broken.

Boot environment menu is currently only available for GRUB. More info see [https://gitlab.com/m_zhou/bieaz bieaz boot environment manager readme].

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select default option {{ic|1=disk=none}} at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}
Depending on your application, separate datasets need to be created for folders inside {{ic|/var/lib}}(not itself!)

Here we create several folders for persistent (shared) data, like we just did for {{ic|/home}}.
d='libvirt lxc docker'
for i in d; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done
{{ic|lxc}} is for Linux container, {{ic|libvirt}} is for storing virtual machine images, etc.

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi
For multi-disk setup, a cron job needs to be configured to sync contents. It should be similar to [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Multi-ESP this article].

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in [#GRUB fixes].
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from BusyBox.
See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from BusyBox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
with
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
And you must install
apk add util-linux
since {{ic|blkid}} from BusyBox does not support ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76/diffs this merge request].
== Enable persistent device names ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77/diffs this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd
Root account is accessed via {{ic|su}} command with root password.

Optionally install {{ic|sudo}} to disable root password and use user's own password instead.

= Boot environment manager =
[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request].

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
== Barebone ==
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K
== Wayland-based lightweight desktop ==
This setup is based on Sway Window Manager and Qt apps.

Encrypted swap
apk add cryptsetup
Sway Window Manager and basic utilities
apk add sway swayidle swaylock grim i3status
Terminal
apk add alacritty
Sound
apk add alsa-utils
Utilities
apk add vim mutt isync lynx git p7zip proxychains-ng
Qt-based desktop environment, with dark theme, fdo keyring, file manager and PDF viewer
apk add qt5-qtwayland kvantum keepassxc pcmanfm zathura-pdf-poppler
Play videos with hardware accelerated decoding
apk add mpv youtube-dl libva-intel-driver
Firefox
apk add firefox-esr
Add MTP (connect to Android phones) and samba support to file manager
apk add gvfs-smb gvfs-mtp
Add dark GTK theme (Adwaita-dark), HiDPI mouse cursor for Sway, GTK icons
apk add gnome-themes-extra
Stat
*rpool used 1.11G
*bpool used 26.6M

= Recovery in Live environment =
Boot Live environment (extended release) and install packages:
setup-alpine # basic settings: keyboard layout, timezone ...
apk-add zfs eudev # zfs-utils and persistent device name support
setup-udev # populate persistent names
modprobe zfs # load kernel module
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-04T08:59:31Z

R3: /* Optional: Enable encrypted swap partition */ fix path

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. BusyBox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

== DO NOT set bootfs property! ==
Do not set {{ic|bootfs}} on any pool!

It will override {{ic|1=root=ZFS=rpool/ROOT/dataset}} kernel parameter and render boot environment menu in GRUB '''INVALID'''.

As GRUB support of ZFS is read-only, you will need to boot into live environment to unset this property if `bootfs` dataset is broken.

Boot environment menu is currently only available for GRUB. More info see [https://gitlab.com/m_zhou/bieaz bieaz boot environment manager readme].

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select default option {{ic|1=disk=none}} at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}
Depending on your application, separate datasets need to be created for folders inside {{ic|/var/lib}}(not itself!)

Here we create several folders for persistent (shared) data, like we just did for {{ic|/home}}.
d='libvirt lxc docker'
for i in d; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done
{{ic|lxc}} is for Linux container, {{ic|libvirt}} is for storing virtual machine images, etc.

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi
For multi-disk setup, a cron job needs to be configured to sync contents. It should be similar to [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Multi-ESP this article].

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in [#GRUB fixes].
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from BusyBox.
See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from BusyBox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
with
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
And you must install
apk add util-linux
since {{ic|blkid}} from BusyBox does not support ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76/diffs this merge request].
== Enable persistent device names ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77/diffs this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd
Root account is accessed via {{ic|su}} command with root password.

Optionally install {{ic|sudo}} to disable root password and use user's own password instead.

= Boot environment manager =
[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request].

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot Live environment (extended release) and install packages:
setup-alpine # basic settings: keyboard layout, timezone ...
apk-add zfs eudev # zfs-utils and persistent device name support
setup-udev # populate persistent names
modprobe zfs # load kernel module
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Alpine configuration management scripts

2021-01-04T08:31:06Z

R3: source from alpine conf

Feature descriptions for available Alpine Linux setup scripts ({{Path|/sbin/setup-*}}).

These script are from alpine-conf package.

(Some particular example usages can be seen at [[Alpine_newbie_install_manual#Ways_to_install_Alpine_into_machines_or_virtuals|Alpine for new users install manuals]].)

== setup-alpine ==

This is the main Alpine configuration and installation script.

The script interactively walks the user through executing several auxiliary <code>setup-*</code> scripts, in the order shown below.

The bracketed options represent example configuration choices, formatted as they may be supplied when manually calling the auxiliary setup scripts, or using a <code>setup-alpine</code> "answerfile" (see below).

# <code>setup-keymap</code> [us us]
# [[#setup-hostname|setup-hostname]] [-n alpine-test]
# [[#setup-interfaces|setup-interfaces]] [-i < interfaces-file]
# <code>/etc/init.d/networking --quiet start &</code>
# if none of the networking interfaces were configured using dhcp, then: <code>[[#setup-dns|setup-dns]]</code> [-d example.com -n "192.168.0.1 [...]"]
# set the root password
# if not in quick mode, then: <code>[[#setup-timezone|setup-timezone]]</code> [-z UTC | -z America/New_York | -p EST+5]
# enable the new hostname (<code>/etc/init.d/hostname --quiet restart</code>)
# add <code>networking</code> and <code>urandom</code> to the '''boot''' rc level, and <code>acpid</code> and <code>cron</code> to the '''default''' rc level, and start the '''boot''' and '''default''' rc services
# extract the fully-qualified domain name and hostname from {{Path|/etc/resolv.conf}} and <code>hostname</code>, and update {{Path|/etc/hosts}}
# <code>[[#setup-proxy|setup-proxy]]</code> [-q <nowiki>"http://webproxy:8080"</nowiki>], and activate proxy if it was configured
# <code>setup-apkrepos</code> [-r (to select a mirror randomly)]
# if not in quick mode, then: <code>[[#setup-sshd|setup-sshd]]</code> [-c openssh | dropbear | none]
# if not in quick mode, then: <code>setup-ntp</code> [-c chrony | openntpd | busybox | none]
# if not in quick mode, then: <code>DEFAULT_DISK=none</code> <code>[[#setup-disk|setup-disk]]</code> <code>-q</code> [-m data /dev/sda] (see [[Installation#Installation_Overview]] about the disk modes)
# if installation mode selected during setup-disk was "data" instead of "sys", then: <code>setup-lbu</code> [/media/sdb1]
# if installation mode selected during setup-disk was "data" instead of "sys", then: <code>setup-apkcache</code> [/media/sdb1/cache | none]

<code>setup-alpine</code> itself accepts the following command-line switches

{{Define|-h|Shows the up-to-date usage help message.}}

{{Define|-a|Create an overlay file: this creates a temporary directory and saves its location in ROOT; however, the script doesn't export this variable so I think this feature isn't currently functional.}}
;-c <var>answerfile</var>
:Create a new "answerfile", with default choices. You can edit the file and then invoke <code>setup-alpine -f <var>answerfile</var></code>.
;-f <var>answerfile</var>
:Use an existing "answerfile", which may override some or all of the interactive prompts.
{{Define|-q|Run in "quick mode".}}

== setup-hostname ==

:<code>setup-hostname</code> [-h] [-n hostname]

Options:

'''-h''' <var>Show help</var>

'''-n''' <var>Specify hostname</var>

This script allows quick and easy setup of the system hostname by writing it to {{Path|/etc/hostname}}. The script prevents you from writing an invalid hostname (such as one that used invalid characters or starts with a '-' or is too long).
The script can be invoked manually or is called as part of the <code>setup-alpine</code> script.

== setup-interfaces ==
{{Cmd|setup-interfaces [-i < <var>interfaces-file</var>]}}

Note that the contents of <var>interfaces-file</var> has to be supplied as stdin, rather than naming the file as an additional argument. The contents should have the format of {{Path|/etc/network/interfaces}}, such as:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp
hostname alpine-test

== setup-dns ==

:<code>setup-dns</code> [-h] [-d domain name] [-n name server]

Options:

'''-h''' <var>Show help</var>

'''-d''' <var>specify search domain name</var>

'''-n''' <var>name server IP</var>

The setup-dns script is stored in {{Path|/sbin/setup-dns}} and allows quick and simple setup of DNS servers (and a DNS search domain if required). Simply running <code>setup-dns</code> will allow interactive use of the script, or the options can be specified.

The information fed to this script is written to {{Path|/etc/resolv.conf}}

Example usage: {{Cmd|setup-dns -d example.org -n 8.8.8.8}}

Example {{Path|/etc/resolv.conf}}:

search example.org
nameserver 8.8.8.8

It can be run manually but is also invoked in the <code>setup-alpine</code> script unless interfaces are configured for DHCP.

== setup-timezone ==
:<code>setup-timezone</code> [-z UTC | -z America/New_York | -p EST+5]

Can pre-select the timezone using either of these switches:

'''-z''' <var>subfolder of</var> {{Path|/usr/share/zoneinfo}}

'''-p''' <var>POSIX TZ format</var>

== setup-proxy ==
:<code>setup-proxy</code> [-hq] [PROXYURL]

Options:

'''-h''' <var>Show help</var>

'''-q''' <var>Quiet mode</var> prevents changes from taking effect until after reboot

This script requests the system proxy to use in the form <code>http://<proxyurl>:<port></code> for example:
<code>http://10.0.0.1:8080</code>

To set no system proxy use <code>none</code>.
This script exports the following environmental variables:

<code>http_proxy=$proxyurl</code>

<code>https_proxy=$proxyurl</code>

<code>ftp_proxy=$proxyurl</code>

where <code>$proxyurl</code> is the value input.
If <code>none</code> was chosen then the value it is set to a blank value (and so no proxy is used).

== setup-sshd ==

:<code>setup-sshd</code> [-h] [-c choice of SSH daemon]

Options:

'''-h''' <var>Show help</var>

'''-c''' <var>SSH daemon</var> where SSH daemon can be one of the following:

<code>openssh</code> install the {{Pkg|openSSH}} daemon

<code>dropbear</code> install the {{Pkg|dropbear}} daemon

<code>none</code> Do not install an SSH daemon

Example usage: {{Cmd|setup-sshd -c dropbear}}

The setup-sshd script is stored in {{Path|/sbin/setup-sshd}} and allows quick and simple setup of either the OpenSSH or Dropbear SSH daemon & client.
It can be run manually but is also invoked in the <code>setup-alpine</code> script.

== setup-apkrepos ==
:<code>setup-apkrepos</code> [-fhr] [REPO...]

Setup <code>apk</code> repositories.

options:

'''-f''' <var>Detect and add fastest mirror</var>

'''-r''' <var>Add a random mirror and do not prompt</var>

'''-1''' <var>Add first mirror on the list (normally a CDN)</var>

This is run as part of the <code>setup-alpine</code> script.

== setup-disk ==

:<code>DEFAULT_DISK=none setup-disk -q</code> [-m data | sys] [<var>mountpoint directory</var> | /dev/sda ...]

In "sys" mode, it's an installer, it permanently installs Alpine on the disk, in "data" mode, it provides a larger and persistent /var volume.

This script accepts the following command-line switches:

;-k <var>kernel flavor</var>
;-o <var>apkovl file</var>
:Restore system from <var>apkovl file</var>
;-m data | sys
:Don't prompt for installation mode. With '''-m data''', the supplied devices are formatted to use as a {{Path|/var}} volume.
{{Define|-r|Use RAID1 with a single disk (degraded mode)}}
{{Define|-L|Create and use volumes in a LVM group}}
;-s <var>swap size in MB</var>
:Use 0 to disable swap
{{Define|-q|Exit quietly if no disks are found}}
{{Define|-v|Verbose mode}}

The script also honors the following environment variables:

<code>BOOT_SIZE</code>
:Size of the boot partition in MB; defaults to 100. Only used if '''-m sys''' is specified or interactively selected.

<code>SWAP_SIZE</code>
:Size of the swap volume in MB; set to 0 to disable swap. If not specified, will default to twice RAM, up to 4096, but won't be more than 1/3 the size of the smallest disk, and if less than 64 will just be 0. Only used if '''-m sys''' is specified or interactively selected.

<code>ROOTFS</code>
:Filesystem to use for the / volume; defaults to ext4. Only used if '''-m sys''' is specified or interactively selected. Supported filesystems are: ext2 ext3 ext4 btrfs xfs.

<code>BOOTFS</code>
:Filesystem to use for the /boot volume; defaults to ext4. Only used if '''-m sys''' is specified or interactively selected. Supported filesystems are: ext2 ext3 ext4 btrfs xfs.

<code>VARFS</code>
:Filesystem to use for the /var volume; defaults to ext4. Only used if '''-m data''' is specified or interactively selected. Supported filesystems are: ext2 ext3 ext4 btrfs xfs.

<code>SYSROOT</code>
:Mountpoint to use when creating volumes and doing traditional disk install ('''-m sys'''). Defaults to {{Path|/mnt}}.

<code>MBR</code>
:Path of MBR binary code, defaults to {{Path|/usr/share/syslinux/mbr.bin}}.

<code>BOOTLOADER</code>
:Bootloader to use, defaults to syslinux. Supported bootloaders are: grub syslinux zipl.

<code>DISKLABEL</code>
:Disklabel to use, defaults to dos. Supported disklabels are: dos gpt eckd.



=== Partitioning ===

If you have complex partitioning needs, you can partition, format, and mount your volumes manually, then just supply the root mountpoint to <code>setup-disk</code>. Doing so implicitly behaves as though '''-m sys''' had also been specified.

See [[Setting up disks manually]] for more information.

==== RAID ====
<code>setup-disk</code> will automatically build a RAID array if you supply the '''-r''' switch, or if you specify more than one device. The array will always be [https://en.m.wikipedia.org/wiki/Standard_RAID_levels#RAID_1 RAID1] (and [https://raid.wiki.kernel.org/index.php/RAID_superblock_formats#The_version-0.90_Superblock_Format --metadata=0.90]) for the /boot volumes, but will be [https://en.m.wikipedia.org/wiki/Standard_RAID_levels#RAID_5 RAID5] (and [https://raid.wiki.kernel.org/index.php/RAID_superblock_formats#The_version-1_Superblock_Format --metadata=1.2] for non-boot volumes when 3 or more devices are supplied.

If you instead want to build your RAID array manually, see [[Setting up a software RAID array]]. Then format and mount the disks, and supply the root mountpoint to <code>setup-disk</code>.

==== LVM ====
<code>setup-disk</code> will automatically build and use volumes in a LVM group if you supply the '''-L''' switch. The group and volumes created by the script will have the following names:

* volume group: '''vg0'''
* swap volume: '''lv_swap''' (only created when swap size > 0)
* root volume: '''lv_root''' (only created when '''-m sys''' is specified or interactively selected)
* var volume: '''lv_var''' (only created when '''-m data''' is specified or interactively selected)

The '''lv_var''' or '''lv_root''' volumes are created to occupy all remaining space in the volume group.

If you need to change any of these settings, you can use <code>vgrename</code>, <code>lvrename</code>, <code>lvreduce</code> or <code>lvresize</code>.

If you instead want to build your LVM system manually, see [[Setting up Logical Volumes with LVM]]. Then format and mount the disks, and supply the root mountpoint to <code>setup-disk</code>.


== setup-lbu ==

This script will only be invoked for by <code>setup-alpine</code> when installing <code>data</code> installation types (ramdisk)

In the previous step, we set up the disk used for the swap partition and for mounting <code>/var</code>. Here we set up where <code>lbu commit</code> will store its backup configuration. See [[Alpine local backup]] for more information.

When started, <code>setup-lbu</code> will prompt where to store your data. The options it will prompt for will be taken from the directories found in <code>/media</code> (except for <code>cdrom</code>). [not sure how these are mounted: are they automatically mounted by setup-lbu? Does the user have to manually mount using another tty?]

== setup-apkcache ==

This script will only be invoked for by <code>setup-alpine</code> when installing <code>data</code> installation types (ramdisk)

In the previous steps, we setup the disk partitions, and told Alpine where it can save its configuration for ramdisk installations. Here we tell Alpine where to save the apk files that you will want to persist across boots. The apkcache is where apk stores cached packages, such that the system does not need to download them at each boot, and doesn't have to depend on the network. See [[Local APK cache]] for a detailed explanation.

You should be able to use a partition that you set up in the previous steps.

== setup-bootable ==
This is a standalone script; it's not invoked by <code>setup-alpine</code> but must be run manually.

Its purpose is to create media that boots into tmpfs by copying the contents of an ISO onto a USB key, CF, or similar media.

For a higher-level walkthrough, see [[Create a Bootable USB#Creating_a_bootable_Alpine_Linux_USB_Stick_from_the_command_line|Creating a bootable Alpine Linux USB Stick from the command line]].

This script accepts the following arguments and command-line switches (you can run <code>setup-bootable -h</code> to see a usage message).

{{Cmd|setup-bootable <var>source</var> [<var>dest</var>]}}

The argument <var>source</var> can be a directory or an ISO (will be mounted to <code>MNT</code> or {{Path|/mnt}}) or a URL (will be downloaded with <code>WGET</code> or <code>wget</code>). The argument <var>dest</var> can be a directory mountpoint, or will default to {{Path|/media/usb}} if not supplied.

{{Define|-k|Keep alpine_dev in {{Path|syslinux.cfg}}; otherwise, replace with UUID.}}
{{Define|-u|Upgrade mode: keep existing {{Path|syslinux.cfg}} and don't run <code>syslinux</code>}}
{{Define|-f|Overwrite {{Path|syslinux.cfg}} even if '''-u''' was specified.}}
{{Define|-s|Force the running of <code>syslinux</code> even if '''-u''' was specified.}}
{{Define|-v|Verbose mode}}

The script will ensure that <var>source</var> and <var>dest</var> are available; will copy the contents of <var>source</var> to <var>dest</var>, ensuring first that there's enough space; and unless '''-u''' was specified, will make <var>dest</var> bootable.



== setup-xorg-base ==
This is a standalone script; it's not invoked by <code>setup-alpine</code> but must be run manually.

Installs a basic xorg configuration, e.g. among other packages: <code>xorg-server xf86-video-vesa xf86-input-evdev xf86-input-mouse xf86-input-keyboard udev</code>.

Additional packages may be supplied as arguments to <code>setup-xorg-base</code>. You might need, for example, some of: <code>xf86-input-synaptics xf86-video-<var>something</var> xinit</code>. For Qemu, see [[Qemu#Using_Xorg_inside_Qemu|Qemu]]. For Intel GPUs, see [[Intel Video]].

== Documentation needed ==

=== setup-xen-dom0 ===

=== setup-gparted-desktop ===
Uses openbox.

This is a standalone script; it's not invoked by <code>setup-alpine</code> but must be run manually.

=== setup-mta ===
Uses ssmtp.

This is a standalone script; it's not invoked by <code>setup-alpine</code> but must be run manually.

=== setup-acf ===
This is a standalone script; it's not invoked by <code>setup-alpine</code> but must be run manually.

This script was named <code>setup-webconf</code> before Alpine 1.9 beta 4.

See [[:Category:ACF|ACF pages]] for more information.

=== setup-ntp ===

= =
* [https://docs.alpinelinux.org/ beta.docs.alpinelinux.org]

[[Category:Installation]]

Root on ZFS with native encryption

2021-01-04T08:27:33Z

R3: /* Format and mount EFI partition */ multidisk

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. BusyBox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

== DO NOT set bootfs property! ==
Do not set {{ic|bootfs}} on any pool!

It will override {{ic|1=root=ZFS=rpool/ROOT/dataset}} kernel parameter and render boot environment menu in GRUB '''INVALID'''.

As GRUB support of ZFS is read-only, you will need to boot into live environment to unset this property if `bootfs` dataset is broken.

Boot environment menu is currently only available for GRUB. More info see [https://gitlab.com/m_zhou/bieaz bieaz boot environment manager readme].

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select default option {{ic|1=disk=none}} at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}
Depending on your application, separate datasets need to be created for folders inside {{ic|/var/lib}}(not itself!)

Here we create several folders for persistent (shared) data, like we just did for {{ic|/home}}.
d='libvirt lxc docker'
for i in d; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done
{{ic|lxc}} is for Linux container, {{ic|libvirt}} is for storing virtual machine images, etc.

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi
For multi-disk setup, a cron job needs to be configured to sync contents. It should be similar to [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Multi-ESP this article].

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in [#GRUB fixes].
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from BusyBox.
See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from BusyBox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
with
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
And you must install
apk add util-linux
since {{ic|blkid}} from BusyBox does not support ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76/diffs this merge request].
== Enable persistent device names ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77/diffs this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd
Root account is accessed via {{ic|su}} command with root password.

Optionally install {{ic|sudo}} to disable root password and use user's own password instead.

= Boot environment manager =
[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request].

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot Live environment (extended release) and install packages:
setup-alpine # basic settings: keyboard layout, timezone ...
apk-add zfs eudev # zfs-utils and persistent device name support
setup-udev # populate persistent names
modprobe zfs # load kernel module
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-04T08:24:08Z

R3: /* Dataset creation */ different example

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. BusyBox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

== DO NOT set bootfs property! ==
Do not set {{ic|bootfs}} on any pool!

It will override {{ic|1=root=ZFS=rpool/ROOT/dataset}} kernel parameter and render boot environment menu in GRUB '''INVALID'''.

As GRUB support of ZFS is read-only, you will need to boot into live environment to unset this property if `bootfs` dataset is broken.

Boot environment menu is currently only available for GRUB. More info see [https://gitlab.com/m_zhou/bieaz bieaz boot environment manager readme].

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select default option {{ic|1=disk=none}} at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}
Depending on your application, separate datasets need to be created for folders inside {{ic|/var/lib}}(not itself!)

Here we create several folders for persistent (shared) data, like we just did for {{ic|/home}}.
d='libvirt lxc docker'
for i in d; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done
{{ic|lxc}} is for Linux container, {{ic|libvirt}} is for storing virtual machine images, etc.

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in [#GRUB fixes].
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from BusyBox.
See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from BusyBox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
with
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
And you must install
apk add util-linux
since {{ic|blkid}} from BusyBox does not support ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76/diffs this merge request].
== Enable persistent device names ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77/diffs this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd
Root account is accessed via {{ic|su}} command with root password.

Optionally install {{ic|sudo}} to disable root password and use user's own password instead.

= Boot environment manager =
[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request].

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot Live environment (extended release) and install packages:
setup-alpine # basic settings: keyboard layout, timezone ...
apk-add zfs eudev # zfs-utils and persistent device name support
setup-udev # populate persistent names
modprobe zfs # load kernel module
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-04T08:22:50Z

R3: /* Dataset creation */ varlib

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. BusyBox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

== DO NOT set bootfs property! ==
Do not set {{ic|bootfs}} on any pool!

It will override {{ic|1=root=ZFS=rpool/ROOT/dataset}} kernel parameter and render boot environment menu in GRUB '''INVALID'''.

As GRUB support of ZFS is read-only, you will need to boot into live environment to unset this property if `bootfs` dataset is broken.

Boot environment menu is currently only available for GRUB. More info see [https://gitlab.com/m_zhou/bieaz bieaz boot environment manager readme].

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select default option {{ic|1=disk=none}} at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}
Depending on your application, separate datasets need to be created for folders inside {{ic|/var/lib}}(not itself!)

Here we create several folders for persistent (shared) data, like we just did for {{ic|/home}}.
for i in {AccountsService,NetworkManager,libvirt,lxc,docker}; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done
{{ic|AccountsService}} is for GNOME desktop. {{ic|libvirt}} is for storing virtual machine images, etc.

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in [#GRUB fixes].
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from BusyBox.
See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from BusyBox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
with
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
And you must install
apk add util-linux
since {{ic|blkid}} from BusyBox does not support ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76/diffs this merge request].
== Enable persistent device names ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77/diffs this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd
Root account is accessed via {{ic|su}} command with root password.

Optionally install {{ic|sudo}} to disable root password and use user's own password instead.

= Boot environment manager =
[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request].

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot Live environment (extended release) and install packages:
setup-alpine # basic settings: keyboard layout, timezone ...
apk-add zfs eudev # zfs-utils and persistent device name support
setup-udev # populate persistent names
modprobe zfs # load kernel module
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-04T07:56:19Z

R3: no-bootfs

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. BusyBox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

== DO NOT set bootfs property! ==
Do not set {{ic|bootfs}} on any pool!

It will override {{ic|1=root=ZFS=rpool/ROOT/dataset}} kernel parameter and render boot environment menu in GRUB '''INVALID'''.

As GRUB support of ZFS is read-only, you will need to boot into live environment to unset this property if `bootfs` dataset is broken.

Boot environment menu is currently only available for GRUB. More info see [https://gitlab.com/m_zhou/bieaz bieaz boot environment manager readme].

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select default option {{ic|1=disk=none}} at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in [#GRUB fixes].
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from BusyBox.
See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from BusyBox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
with
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
And you must install
apk add util-linux
since {{ic|blkid}} from BusyBox does not support ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76/diffs this merge request].
== Enable persistent device names ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77/diffs this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd
Root account is accessed via {{ic|su}} command with root password.

Optionally install {{ic|sudo}} to disable root password and use user's own password instead.

= Boot environment manager =
[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request].

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot Live environment (extended release) and install packages:
setup-alpine # basic settings: keyboard layout, timezone ...
apk-add zfs eudev # zfs-utils and persistent device name support
setup-udev # populate persistent names
modprobe zfs # load kernel module
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-04T04:01:09Z

R3: /* Boot environment manager */ aports

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. BusyBox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select default option {{ic|1=disk=none}} at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in [#GRUB fixes].
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from BusyBox.
See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from BusyBox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
with
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
And you must install
apk add util-linux
since {{ic|blkid}} from BusyBox does not support ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76/diffs this merge request].
== Enable persistent device names ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77/diffs this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd
Root account is accessed via {{ic|su}} command with root password.

Optionally install {{ic|sudo}} to disable root password and use user's own password instead.

= Boot environment manager =
[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request].

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot Live environment (extended release) and install packages:
setup-alpine # basic settings: keyboard layout, timezone ...
apk-add zfs eudev # zfs-utils and persistent device name support
setup-udev # populate persistent names
modprobe zfs # load kernel module
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-03T19:07:18Z

R3: /* Boot environment manager */ denpendent

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. BusyBox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select default option {{ic|1=disk=none}} at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in [#GRUB fixes].
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from BusyBox.
See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from BusyBox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
with
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
And you must install
apk add util-linux
since {{ic|blkid}} from BusyBox does not support ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76/diffs this merge request].
== Enable persistent device names ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77/diffs this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd
Root account is accessed via {{ic|su}} command with root password.

Optionally install {{ic|sudo}} to disable root password and use user's own password instead.

= Boot environment manager =
[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It hasn't been packaged for Alpine Linux. To install, download the latest release, extract the files and move scripts to directories defined in Makefile.

It depends on
apk add coreutils

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot Live environment (extended release) and install packages:
setup-alpine # basic settings: keyboard layout, timezone ...
apk-add zfs eudev # zfs-utils and persistent device name support
setup-udev # populate persistent names
modprobe zfs # load kernel module
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-03T18:30:16Z

R3: /* Add normal user account */ be manager

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. BusyBox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select default option {{ic|1=disk=none}} at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in [#GRUB fixes].
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from BusyBox.
See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from BusyBox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
with
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
And you must install
apk add util-linux
since {{ic|blkid}} from BusyBox does not support ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76/diffs this merge request].
== Enable persistent device names ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77/diffs this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd
Root account is accessed via {{ic|su}} command with root password.

Optionally install {{ic|sudo}} to disable root password and use user's own password instead.

= Boot environment manager =
[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.

It hasn't been packaged for Alpine Linux. To install, download the latest release, extract the files and move scripts to directories defined in Makefile.

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot Live environment (extended release) and install packages:
setup-alpine # basic settings: keyboard layout, timezone ...
apk-add zfs eudev # zfs-utils and persistent device name support
setup-udev # populate persistent names
modprobe zfs # load kernel module
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-03T16:25:51Z

R3: /* Add normal user account */ sudo

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. BusyBox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select default option {{ic|1=disk=none}} at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in [#GRUB fixes].
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from BusyBox.
See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from BusyBox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
with
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
And you must install
apk add util-linux
since {{ic|blkid}} from BusyBox does not support ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76/diffs this merge request].
== Enable persistent device names ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77/diffs this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd
Root account is accessed via {{ic|su}} command with root password.

Optionally install {{ic|sudo}} to disable root password and use user's own password instead.

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot Live environment (extended release) and install packages:
setup-alpine # basic settings: keyboard layout, timezone ...
apk-add zfs eudev # zfs-utils and persistent device name support
setup-udev # populate persistent names
modprobe zfs # load kernel module
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-03T16:02:08Z

R3: /* Recovery in Live environment */ desc

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. BusyBox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select default option {{ic|1=disk=none}} at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in [#GRUB fixes].
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from BusyBox.
See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from BusyBox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
with
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
And you must install
apk add util-linux
since {{ic|blkid}} from BusyBox does not support ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76/diffs this merge request].
== Enable persistent device names ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77/diffs this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot Live environment (extended release) and install packages:
setup-alpine # basic settings: keyboard layout, timezone ...
apk-add zfs eudev # zfs-utils and persistent device name support
setup-udev # populate persistent names
modprobe zfs # load kernel module
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-03T15:59:01Z

R3: /* Setup live environment */ wording

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. BusyBox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select default option {{ic|1=disk=none}} at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in [#GRUB fixes].
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from BusyBox.
See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from BusyBox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
with
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
And you must install
apk add util-linux
since {{ic|blkid}} from BusyBox does not support ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76/diffs this merge request].
== Enable persistent device names ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77/diffs this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot extra and install packages:
setup-alpine
apk-add zfs eudev
setup-udev
modprobe zfs
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-03T15:57:45Z

R3: /* Initramfs fixes */ fixes

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. BusyBox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in [#GRUB fixes].
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from BusyBox.
See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from BusyBox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
with
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
And you must install
apk add util-linux
since {{ic|blkid}} from BusyBox does not support ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76/diffs this merge request].
== Enable persistent device names ==
Apply fixes in [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77/diffs this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot extra and install packages:
setup-alpine
apk-add zfs eudev
setup-udev
modprobe zfs
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-03T15:55:43Z

R3: /* Finish GRUB installation */ typo

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. BusyBox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in [#GRUB fixes].
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from BusyBox.
See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from BusyBox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
with
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
And you must install
apk add util-linux
since {{ic|blkid}} from BusyBox does not support ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76].
== Enable persistent device names ==
Special modifications need to be made to populate {{ic|/dev/disk/by-*}} in initramfs.

See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot extra and install packages:
setup-alpine
apk-add zfs eudev
setup-udev
modprobe zfs
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-03T13:27:45Z

R3: use correct name BusyBox

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. BusyBox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in WARNING.
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from BusyBox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from BusyBox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
with
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
And you must install
apk add util-linux
since {{ic|blkid}} from BusyBox does not support ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76].
== Enable persistent device names ==
Special modifications need to be made to populate {{ic|/dev/disk/by-*}} in initramfs.

See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot extra and install packages:
setup-alpine
apk-add zfs eudev
setup-udev
modprobe zfs
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-03T13:26:27Z

R3: /* Missing root pool */ update

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in WARNING.
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.
[https://lists.gnu.org/archive/html/grub-devel/2021-01/msg00003.html This patch] will warn about failed detection and allow customized detection method.

Before the patch is merged, I recommend to replace the following in {{ic|/etc/grub.d/10_linux}}
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
with
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
And you must install
apk add util-linux
since {{ic|blkid}} from BusyBox does not support ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76].
== Enable persistent device names ==
Special modifications need to be made to populate {{ic|/dev/disk/by-*}} in initramfs.

See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot extra and install packages:
setup-alpine
apk-add zfs eudev
setup-udev
modprobe zfs
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-02T18:38:53Z

R3: /* Install system utilities */ details

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.

Now run the following command to populate persistent device names in live system:
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in WARNING.
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.

More detail see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html].

As the pool name is stored as disk label, it is possible to probe disk label and use that as root pool name,

As the pool name is stored as disk label, it is possible to probe file system label and use that as root pool name

First, install util-linux:
apk add util-linux
blkid from Busybox does not support ZFS filesystem.

Replace {{ic|/etc/grub.d/10_linux}} with underline
fi;;
xzfs)
# ZFS pool name is stored as file system label
# blkid from util-linux
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
bootfs="`make_system_path_relative_to_its_root / | sed -e "s,@$,,"`"
Or with lsblk, also from util-linux
rpool=`lsblk -no LABEL ${GRUB_DEVICE}`
Note that some version of blkid, such as the one shipped with busybox, will return an empty result because of missing support of ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76].
== Enable persistent device names ==
Special modifications need to be made to populate {{ic|/dev/disk/by-*}} in initramfs.

See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot extra and install packages:
setup-alpine
apk-add zfs eudev
setup-udev
modprobe zfs
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-01T14:26:25Z

R3: /* Finish GRUB installation */ reload

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Reload
source /etc/profile
Apply fixes in WARNING.
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.

More detail see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html].

As the pool name is stored as disk label, it is possible to probe disk label and use that as root pool name,

As the pool name is stored as disk label, it is possible to probe file system label and use that as root pool name

First, install util-linux:
apk add util-linux
blkid from Busybox does not support ZFS filesystem.

Replace {{ic|/etc/grub.d/10_linux}} with underline
fi;;
xzfs)
# ZFS pool name is stored as file system label
# blkid from util-linux
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
bootfs="`make_system_path_relative_to_its_root / | sed -e "s,@$,,"`"
Or with lsblk, also from util-linux
rpool=`lsblk -no LABEL ${GRUB_DEVICE}`
Note that some version of blkid, such as the one shipped with busybox, will return an empty result because of missing support of ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76].
== Enable persistent device names ==
Special modifications need to be made to populate {{ic|/dev/disk/by-*}} in initramfs.

See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot extra and install packages:
setup-alpine
apk-add zfs eudev
setup-udev
modprobe zfs
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-01T14:25:54Z

R3: /* Finish GRUB installation */ add to profile

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile
Apply fixes in WARNING.
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.

More detail see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html].

As the pool name is stored as disk label, it is possible to probe disk label and use that as root pool name,

As the pool name is stored as disk label, it is possible to probe file system label and use that as root pool name

First, install util-linux:
apk add util-linux
blkid from Busybox does not support ZFS filesystem.

Replace {{ic|/etc/grub.d/10_linux}} with underline
fi;;
xzfs)
# ZFS pool name is stored as file system label
# blkid from util-linux
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
bootfs="`make_system_path_relative_to_its_root / | sed -e "s,@$,,"`"
Or with lsblk, also from util-linux
rpool=`lsblk -no LABEL ${GRUB_DEVICE}`
Note that some version of blkid, such as the one shipped with busybox, will return an empty result because of missing support of ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76].
== Enable persistent device names ==
Special modifications need to be made to populate {{ic|/dev/disk/by-*}} in initramfs.

See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot extra and install packages:
setup-alpine
apk-add zfs eudev
setup-udev
modprobe zfs
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-01T14:21:20Z

R3: order

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.

More detail see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html].

As the pool name is stored as disk label, it is possible to probe disk label and use that as root pool name,

As the pool name is stored as disk label, it is possible to probe file system label and use that as root pool name

First, install util-linux:
apk add util-linux
blkid from Busybox does not support ZFS filesystem.

Replace {{ic|/etc/grub.d/10_linux}} with underline
fi;;
xzfs)
# ZFS pool name is stored as file system label
# blkid from util-linux
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
bootfs="`make_system_path_relative_to_its_root / | sed -e "s,@$,,"`"
Or with lsblk, also from util-linux
rpool=`lsblk -no LABEL ${GRUB_DEVICE}`
Note that some version of blkid, such as the one shipped with busybox, will return an empty result because of missing support of ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Initramfs fixes =
== Fix zfs decrypt ==
See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76].
== Enable persistent device names ==
Special modifications need to be made to populate {{ic|/dev/disk/by-*}} in initramfs.

See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot extra and install packages:
setup-alpine
apk-add zfs eudev
setup-udev
modprobe zfs
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-01T02:35:29Z

R3: /* Missing root pool */ rm

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.

More detail see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html].

As the pool name is stored as disk label, it is possible to probe disk label and use that as root pool name,

As the pool name is stored as disk label, it is possible to probe file system label and use that as root pool name

First, install util-linux:
apk add util-linux
blkid from Busybox does not support ZFS filesystem.

Replace {{ic|/etc/grub.d/10_linux}} with underline
fi;;
xzfs)
# ZFS pool name is stored as file system label
# blkid from util-linux
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
bootfs="`make_system_path_relative_to_its_root / | sed -e "s,@$,,"`"
Or with lsblk, also from util-linux
rpool=`lsblk -no LABEL ${GRUB_DEVICE}`
Note that some version of blkid, such as the one shipped with busybox, will return an empty result because of missing support of ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Initramfs fixes =
== Fix zfs decrypt ==
See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76].
== Enable persistent device names ==
Special modifications need to be made to populate {{ic|/dev/disk/by-*}} in initramfs.

See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot extra and install packages:
setup-alpine
apk-add zfs eudev
setup-udev
modprobe zfs
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2021-01-01T02:33:57Z

R3: /* Missing root pool */ blkid

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.

More detail see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html].

A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

This workaround uses {{ic|zdb}}, which does not have a stable output, according to manual page.
zdb -l ${GRUB_DEVICE} | awk -F \' '/ name/ { print $2 }'

As the pool name is stored as disk label, it is possible to probe disk label and use that as root pool name,

An alternative using blkid from util-linux is:
apk add util-linux
As the pool name is stored as disk label, it is possible to probe file system label and use that as root pool name

Replace {{ic|/etc/grub.d/10_linux}} with underline
fi;;
xzfs)
# ZFS pool name is stored as file system label
# blkid from util-linux
rpool=`blkid -s LABEL -o value ${GRUB_DEVICE}`
bootfs="`make_system_path_relative_to_its_root / | sed -e "s,@$,,"`"
Or with lsblk, also from util-linux
rpool=`lsblk -no LABEL ${GRUB_DEVICE}`
Note that some version of blkid, such as the one shipped with busybox, will return an empty result because of missing support of ZFS.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Initramfs fixes =
== Fix zfs decrypt ==
See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76].
== Enable persistent device names ==
Special modifications need to be made to populate {{ic|/dev/disk/by-*}} in initramfs.

See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot extra and install packages:
setup-alpine
apk-add zfs eudev
setup-udev
modprobe zfs
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-31T18:12:58Z

R3: /* Optional: Enable encrypted swap partition */ eudev

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html].

A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

This workaround uses {{ic|zdb}}, which does not have a stable output, according to manual page.
zdb -l ${GRUB_DEVICE} | awk -F \' '/ name/ { print $2 }'

As the pool name is stored as disk label, it is possible to probe disk label and use that as root pool name,

An alternative using blkid from util-linux is:
apk add util-linux
And replace {{ic|/etc/grub.d/10_linux}} with underline
fi;;
xzfs)
rpool=`eval "$(blkid info -o export ${GRUB_DEVICE})" && echo $LABEL`
bootfs="`make_system_path_relative_to_its_root / | sed -e "s,@$,,"`"
Busybox version of blkid will return an empty result.

Choose one to your liking and don't forget to apply this every GRUB update.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Initramfs fixes =
== Fix zfs decrypt ==
See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76].
== Enable persistent device names ==
Special modifications need to be made to populate {{ic|/dev/disk/by-*}} in initramfs.

See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs eudev"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot extra and install packages:
setup-alpine
apk-add zfs eudev
setup-udev
modprobe zfs
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-31T18:12:25Z

R3: /* Importing pools on boot */ details

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html].

A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

This workaround uses {{ic|zdb}}, which does not have a stable output, according to manual page.
zdb -l ${GRUB_DEVICE} | awk -F \' '/ name/ { print $2 }'

As the pool name is stored as disk label, it is possible to probe disk label and use that as root pool name,

An alternative using blkid from util-linux is:
apk add util-linux
And replace {{ic|/etc/grub.d/10_linux}} with underline
fi;;
xzfs)
rpool=`eval "$(blkid info -o export ${GRUB_DEVICE})" && echo $LABEL`
bootfs="`make_system_path_relative_to_its_root / | sed -e "s,@$,,"`"
Busybox version of blkid will return an empty result.

Choose one to your liking and don't forget to apply this every GRUB update.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Initramfs fixes =
== Fix zfs decrypt ==
See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76].
== Enable persistent device names ==
Special modifications need to be made to populate {{ic|/dev/disk/by-*}} in initramfs.

See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Importing pools on boot =
{{ic|zpool.cache}} will be added to initramfs and zpool command will import pools contained in this cache.
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot extra and install packages:
setup-alpine
apk-add zfs eudev
setup-udev
modprobe zfs
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-31T17:24:12Z

R3: /* Missing root pool */ file name

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html].

A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

This workaround uses {{ic|zdb}}, which does not have a stable output, according to manual page.
zdb -l ${GRUB_DEVICE} | awk -F \' '/ name/ { print $2 }'

As the pool name is stored as disk label, it is possible to probe disk label and use that as root pool name,

An alternative using blkid from util-linux is:
apk add util-linux
And replace {{ic|/etc/grub.d/10_linux}} with underline
fi;;
xzfs)
rpool=`eval "$(blkid info -o export ${GRUB_DEVICE})" && echo $LABEL`
bootfs="`make_system_path_relative_to_its_root / | sed -e "s,@$,,"`"
Busybox version of blkid will return an empty result.

Choose one to your liking and don't forget to apply this every GRUB update.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Initramfs fixes =
== Fix zfs decrypt ==
See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76].
== Enable persistent device names ==
Special modifications need to be made to populate {{ic|/dev/disk/by-*}} in initramfs.

See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Importing pools on boot =
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot extra and install packages:
setup-alpine
apk-add zfs eudev
setup-udev
modprobe zfs
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-31T17:23:35Z

R3: /* Missing root pool */ procedure

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
setup-udev

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== GRUB fixes ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils
=== Missing root pool ===
2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html].

A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

This workaround uses {{ic|zdb}}, which does not have a stable output, according to manual page.
zdb -l ${GRUB_DEVICE} | awk -F \' '/ name/ { print $2 }'

As the pool name is stored as disk label, it is possible to probe disk label and use that as root pool name,

An alternative using blkid from util-linux is:
apk add util-linux
And replace with underline
fi;;
xzfs)
rpool=`eval "$(blkid info -o export ${GRUB_DEVICE})" && echo $LABEL`
bootfs="`make_system_path_relative_to_its_root / | sed -e "s,@$,,"`"

Busybox version of blkid will return an empty result.

Choose one to your liking and don't forget to apply this every GRUB update.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Initramfs fixes =
== Fix zfs decrypt ==
See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76].
== Enable persistent device names ==
Special modifications need to be made to populate {{ic|/dev/disk/by-*}} in initramfs.

See [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 this merge request].

With the changes in merge request applied, add {{ic|eudev}} to {{ic|/etc/mkinitfs/mkinitfs.conf}}.
sed -i 's|zfs|zfs eudev|' /etc/mkinitfs/mkinitfs.conf
Rebuild initramfs with
mkinitfs $(ls -1 /lib/modules/)

= Mount datasets at boot =
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit # zfs monitoring
Mounting {{ic|/boot}} dataset with fstab need {{ic|1=mountpoint=legacy}}:
umount /boot/efi
zfs set mountpoint=legacy bpool_$poolUUID/BOOT/default
mount /boot
mount /boot/efi

= Importing pools on boot =
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

= Add normal user account =
adduser -s /bin/sh -G wheel -G video -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

reboot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
Boot extra and install packages:
setup-alpine
apk-add zfs eudev
setup-udev
modprobe zfs
Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/''$dataset'' $MOUNTPOINT/boot # legacy mountpoint
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID