Alpine Linux - User contributions [en]

Root on ZFS with native encryption

2020-12-31T04:13:25Z

R2: /* Fix zfs decrypt */ fix

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Initramfs fixes =
== Fix zfs decrypt ==
As of this writing, the initramfs script has a bug for entering ZFS password at boot. When booting the system, root dataset will fail to mount {{ic|sh: `active`, unknown operand}}and drop into emergency shell.

In {{ic|/usr/share/mkinitfs/initramfs-init}}:
# BROKEN: if [ $(zpool list -H -o feature@encryption $_root_pool) = "active" ]; then
# add double quotes around $()
if [ "$(zpool list -H -o feature@encryption $_root_pool)" = "active" ]; then
local _encryption_root=$(zfs get -H -o value encryptionroot $_root_vol)
if [ "$_encryption_root" != "-" ]; then
eval zfs load-key $_encryption_root
fi
fi

== Enable persistent device names ==
Special modifications need to be made to populate {{ic|/dev/disk/by-*}} in initramfs.

Ensure {{ic|eudev}} is installed
apk add eudev
Create {{ic|/etc/mkinitfs/features.d/eudev.files}} to add {{ic|eudev}} to initramfs.
tee /etc/mkinitfs/features.d/eudev.files << EOF
/bin/udevadm
/sbin/udevadm
/sbin/udevd
/etc/udev/*
/lib/udev/*
/usr/lib/libudev*
EOF
Edit {{ic|/usr/share/mkinitfs/initramfs-init}}.

Add functions from {{ic|/etc/init.d/udev*}} at the beginning of the file.
{{Text art|<nowiki>
# persistent device names from eudev, see /etc/init.d/udev*
eudev_start_pre() {
# load unix domain sockets if built as module, Bug #221253
# and not yet loaded, Bug #363549
if [ ! -e /proc/net/unix ]; then
if ! modprobe unix; then
eerror "Cannot load the unix domain socket module"
return 1
fi
fi

if [ -e /proc/sys/kernel/hotplug ]; then
echo "" >/proc/sys/kernel/hotplug
fi

return 0
}

eudev_dir_writeable()
{
touch "$1"/.test.$$ 2>/dev/null && rm "$1"/.test.$$
}

# store persistent-rules that got created while booting
# when / was still read-only
eudev_store_persistent_rules()
{
# create /etc/udev/rules.d if it does not exist and /etc/udev is writable
[ -d /etc/udev/rules.d ] || \
eudev_dir_writeable /etc/udev && \
mkdir -p /etc/udev/rules.d

# only continue if rules-directory is writable
eudev_dir_writeable /etc/udev/rules.d || return 0

local file dest
for file in /run/udev/tmp-rules--*; do
dest=${file##*tmp-rules--}
[ "$dest" = '*' ] && break
type=${dest##70-persistent-}
type=${type%%.rules}
cat "$file" >> /etc/udev/rules.d/"$dest" && rm -f "$file"
done
}

eudev_start()
{
eudev_start_pre
udevd -d
# store persistent-rules that got created while booting
# when / was still read-only
eudev_store_persistent_rules
# Populating /dev with existing devices through uevents"
udevadm trigger --type=subsystems --action=add
udevadm trigger --type=devices --action=add
}
</nowiki>}}
After the following section, which populates {{ic|/dev}}
{{Text art|<nowiki>
mount -t devtmpfs -o exec,nosuid,mode=0755,size=2M devtmpfs /dev 2>/dev/null \
|| mount -t tmpfs -o exec,nosuid,mode=0755,size=2M tmpfs /dev

# pty device nodes (later system will need it)
[ -c /dev/ptmx ] || mknod -m 666 /dev/ptmx c 5 2
[ -d /dev/pts ] || mkdir -m 755 /dev/pts
mount -t devpts -o gid=5,mode=0620,noexec,nosuid devpts /dev/pts

# shared memory area (later system will need it)
[ -d /dev/shm ] || mkdir /dev/shm
mount -t tmpfs -o nodev,nosuid,noexec shm /dev/shm
</nowiki>}}
add this paragraph.
{{Text art|<nowiki>
# persistent device names from eudev
if [ -f /sbin/udevadm ]; then
eudev_start
fi
</nowiki>}}
To let {{ic|zpool import}} to use persistent names, use zpool.cache to import pool
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID
echo /etc/zfs/zpool.cache >> /etc/mkinitfs/features.d/zfs.files
replace {{ic|zpool import}} with {{ic|zpool import -c /etc/zfs/zpool.cache}}
find and replace
-d /dev $_root_pool
with
-c /etc/zfs/zpool.cache
and
-d /dev -f $_root_pool
with
-f -c /etc/zfs/zpool.cache
Rebuild initramfs with
mkinitfs

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.

{{ic|shadow}} is available in community repo. Enable it first:
vi /etc/apk/repositories
# uncomment community line
Install
apk add shadow sudo eudev

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/sh -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

Until this is fixed, we need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs rpool_$UUID/ROOT/default /sysroot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-31T04:00:09Z

R2: /* Enable persistent device names */ find and replace

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Initramfs fixes =
== Fix zfs decrypt ==
As of this writing, the initramfs has a bug for entering ZFS password at boot. When booting the system, root dataset will fail to mount {{ic|sh: `active`, unknown operand}}and drop into emergency shell.

In {{ic|/usr/share/mkinitfs/initramfs-init}}:
# Ask for encryption password
if [ $(zpool list -H -o feature@encryption $_root_pool) = "active" ]; then
# replace underline with true will fix it
# if true; then
local _encryption_root=$(zfs get -H -o value encryptionroot $_root_vol)
if [ "$_encryption_root" != "-" ]; then
eval zfs load-key $_encryption_root
fi
fi

== Enable persistent device names ==
Special modifications need to be made to populate {{ic|/dev/disk/by-*}} in initramfs.

Ensure {{ic|eudev}} is installed
apk add eudev
Create {{ic|/etc/mkinitfs/features.d/eudev.files}} to add {{ic|eudev}} to initramfs.
tee /etc/mkinitfs/features.d/eudev.files << EOF
/bin/udevadm
/sbin/udevadm
/sbin/udevd
/etc/udev/*
/lib/udev/*
/usr/lib/libudev*
EOF
Edit {{ic|/usr/share/mkinitfs/initramfs-init}}.

Add functions from {{ic|/etc/init.d/udev*}} at the beginning of the file.
{{Text art|<nowiki>
# persistent device names from eudev, see /etc/init.d/udev*
eudev_start_pre() {
# load unix domain sockets if built as module, Bug #221253
# and not yet loaded, Bug #363549
if [ ! -e /proc/net/unix ]; then
if ! modprobe unix; then
eerror "Cannot load the unix domain socket module"
return 1
fi
fi

if [ -e /proc/sys/kernel/hotplug ]; then
echo "" >/proc/sys/kernel/hotplug
fi

return 0
}

eudev_dir_writeable()
{
touch "$1"/.test.$$ 2>/dev/null && rm "$1"/.test.$$
}

# store persistent-rules that got created while booting
# when / was still read-only
eudev_store_persistent_rules()
{
# create /etc/udev/rules.d if it does not exist and /etc/udev is writable
[ -d /etc/udev/rules.d ] || \
eudev_dir_writeable /etc/udev && \
mkdir -p /etc/udev/rules.d

# only continue if rules-directory is writable
eudev_dir_writeable /etc/udev/rules.d || return 0

local file dest
for file in /run/udev/tmp-rules--*; do
dest=${file##*tmp-rules--}
[ "$dest" = '*' ] && break
type=${dest##70-persistent-}
type=${type%%.rules}
cat "$file" >> /etc/udev/rules.d/"$dest" && rm -f "$file"
done
}

eudev_start()
{
eudev_start_pre
udevd -d
# store persistent-rules that got created while booting
# when / was still read-only
eudev_store_persistent_rules
# Populating /dev with existing devices through uevents"
udevadm trigger --type=subsystems --action=add
udevadm trigger --type=devices --action=add
}
</nowiki>}}
After the following section, which populates {{ic|/dev}}
{{Text art|<nowiki>
mount -t devtmpfs -o exec,nosuid,mode=0755,size=2M devtmpfs /dev 2>/dev/null \
|| mount -t tmpfs -o exec,nosuid,mode=0755,size=2M tmpfs /dev

# pty device nodes (later system will need it)
[ -c /dev/ptmx ] || mknod -m 666 /dev/ptmx c 5 2
[ -d /dev/pts ] || mkdir -m 755 /dev/pts
mount -t devpts -o gid=5,mode=0620,noexec,nosuid devpts /dev/pts

# shared memory area (later system will need it)
[ -d /dev/shm ] || mkdir /dev/shm
mount -t tmpfs -o nodev,nosuid,noexec shm /dev/shm
</nowiki>}}
add this paragraph.
{{Text art|<nowiki>
# persistent device names from eudev
if [ -f /sbin/udevadm ]; then
eudev_start
fi
</nowiki>}}
To let {{ic|zpool import}} to use persistent names, use zpool.cache to import pool
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID
echo /etc/zfs/zpool.cache >> /etc/mkinitfs/features.d/zfs.files
replace {{ic|zpool import}} with {{ic|zpool import -c /etc/zfs/zpool.cache}}
find and replace
-d /dev $_root_pool
with
-c /etc/zfs/zpool.cache
and
-d /dev -f $_root_pool
with
-f -c /etc/zfs/zpool.cache
Rebuild initramfs with
mkinitfs

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.

{{ic|shadow}} is available in community repo. Enable it first:
vi /etc/apk/repositories
# uncomment community line
Install
apk add shadow sudo eudev

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/sh -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

Until this is fixed, we need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs rpool_$UUID/ROOT/default /sysroot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-31T03:54:19Z

R2: order

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Initramfs fixes =
== Fix zfs decrypt ==
As of this writing, the initramfs has a bug for entering ZFS password at boot. When booting the system, root dataset will fail to mount {{ic|sh: `active`, unknown operand}}and drop into emergency shell.

In {{ic|/usr/share/mkinitfs/initramfs-init}}:
# Ask for encryption password
if [ $(zpool list -H -o feature@encryption $_root_pool) = "active" ]; then
# replace underline with true will fix it
# if true; then
local _encryption_root=$(zfs get -H -o value encryptionroot $_root_vol)
if [ "$_encryption_root" != "-" ]; then
eval zfs load-key $_encryption_root
fi
fi

== Enable persistent device names ==
Special modifications need to be made to populate {{ic|/dev/disk/by-*}} in initramfs.

Ensure {{ic|eudev}} is installed
apk add eudev
Create {{ic|/etc/mkinitfs/features.d/eudev.files}} to add {{ic|eudev}} to initramfs.
tee /etc/mkinitfs/features.d/eudev.files << EOF
/bin/udevadm
/sbin/udevadm
/sbin/udevd
/etc/udev/*
/lib/udev/*
/usr/lib/libudev*
EOF
Edit {{ic|/usr/share/mkinitfs/initramfs-init}}.

Add functions from {{ic|/etc/init.d/udev*}} at the beginning of the file.
{{Text art|<nowiki>
# persistent device names from eudev, see /etc/init.d/udev*
eudev_start_pre() {
# load unix domain sockets if built as module, Bug #221253
# and not yet loaded, Bug #363549
if [ ! -e /proc/net/unix ]; then
if ! modprobe unix; then
eerror "Cannot load the unix domain socket module"
return 1
fi
fi

if [ -e /proc/sys/kernel/hotplug ]; then
echo "" >/proc/sys/kernel/hotplug
fi

return 0
}

eudev_dir_writeable()
{
touch "$1"/.test.$$ 2>/dev/null && rm "$1"/.test.$$
}

# store persistent-rules that got created while booting
# when / was still read-only
eudev_store_persistent_rules()
{
# create /etc/udev/rules.d if it does not exist and /etc/udev is writable
[ -d /etc/udev/rules.d ] || \
eudev_dir_writeable /etc/udev && \
mkdir -p /etc/udev/rules.d

# only continue if rules-directory is writable
eudev_dir_writeable /etc/udev/rules.d || return 0

local file dest
for file in /run/udev/tmp-rules--*; do
dest=${file##*tmp-rules--}
[ "$dest" = '*' ] && break
type=${dest##70-persistent-}
type=${type%%.rules}
cat "$file" >> /etc/udev/rules.d/"$dest" && rm -f "$file"
done
}

eudev_start()
{
eudev_start_pre
udevd -d
# store persistent-rules that got created while booting
# when / was still read-only
eudev_store_persistent_rules
# Populating /dev with existing devices through uevents"
udevadm trigger --type=subsystems --action=add
udevadm trigger --type=devices --action=add
}
</nowiki>}}
After the following section, which populates {{ic|/dev}}
{{Text art|<nowiki>
mount -t devtmpfs -o exec,nosuid,mode=0755,size=2M devtmpfs /dev 2>/dev/null \
|| mount -t tmpfs -o exec,nosuid,mode=0755,size=2M tmpfs /dev

# pty device nodes (later system will need it)
[ -c /dev/ptmx ] || mknod -m 666 /dev/ptmx c 5 2
[ -d /dev/pts ] || mkdir -m 755 /dev/pts
mount -t devpts -o gid=5,mode=0620,noexec,nosuid devpts /dev/pts

# shared memory area (later system will need it)
[ -d /dev/shm ] || mkdir /dev/shm
mount -t tmpfs -o nodev,nosuid,noexec shm /dev/shm
</nowiki>}}
add this paragraph.
{{Text art|<nowiki>
# persistent device names from eudev
if [ -f /sbin/udevadm ]; then
eudev_start
fi
</nowiki>}}
To let {{ic|zpool import}} to use persistent names, use zpool.cache to import pool
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID
echo /etc/zfs/zpool.cache >> /etc/mkinitfs/features.d/zfs.files
replace {{ic|zpool import}} with {{ic|zpool import -c /etc/zfs/zpool.cache}}
sed -i 's|-d /dev|-c /etc/zfs/zpool.cache|g' /usr/share/mkinitfs/initramfs-init
Rebuild initramfs with
mkinitfs

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.

{{ic|shadow}} is available in community repo. Enable it first:
vi /etc/apk/repositories
# uncomment community line
Install
apk add shadow sudo eudev

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/sh -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =

Until this is fixed, we need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs rpool_$UUID/ROOT/default /sysroot
= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-31T03:47:50Z

R2: /* Enable persistent device names in initramfs */ import from cache

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.

{{ic|shadow}} is available in community repo. Enable it first:
vi /etc/apk/repositories
# uncomment community line
Install
apk add shadow sudo eudev

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/sh -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =
As of this writing, the initramfs has a bug for entering ZFS password at boot. When booting the system, root dataset will fail to mount {{ic|sh: `active`, unknown operand}}and drop into emergency shell.

In {{ic|/usr/share/mkinitfs/initramfs-init}}:
# Ask for encryption password
if [ $(zpool list -H -o feature@encryption $_root_pool) = "active" ]; then
# replace underline with true will fix it
# if true; then
local _encryption_root=$(zfs get -H -o value encryptionroot $_root_vol)
if [ "$_encryption_root" != "-" ]; then
eval zfs load-key $_encryption_root
fi
fi

Until this is fixed, we need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs rpool_$UUID/ROOT/default /sysroot
= Enable persistent device names in initramfs =
Special modifications need to be made to populate {{ic|/dev/disk/by-*}} in initramfs.

Ensure {{ic|eudev}} is installed
apk add eudev
Create {{ic|/etc/mkinitfs/features.d/eudev.files}} to add {{ic|eudev}} to initramfs.
tee /etc/mkinitfs/features.d/eudev.files << EOF
/bin/udevadm
/sbin/udevadm
/sbin/udevd
/etc/udev/*
/lib/udev/*
/usr/lib/libudev*
EOF
Edit {{ic|/usr/share/mkinitfs/initramfs-init}}.

Add functions from {{ic|/etc/init.d/udev*}} at the beginning of the file.
{{Text art|<nowiki>
# persistent device names from eudev, see /etc/init.d/udev*
eudev_start_pre() {
# load unix domain sockets if built as module, Bug #221253
# and not yet loaded, Bug #363549
if [ ! -e /proc/net/unix ]; then
if ! modprobe unix; then
eerror "Cannot load the unix domain socket module"
return 1
fi
fi

if [ -e /proc/sys/kernel/hotplug ]; then
echo "" >/proc/sys/kernel/hotplug
fi

return 0
}

eudev_dir_writeable()
{
touch "$1"/.test.$$ 2>/dev/null && rm "$1"/.test.$$
}

# store persistent-rules that got created while booting
# when / was still read-only
eudev_store_persistent_rules()
{
# create /etc/udev/rules.d if it does not exist and /etc/udev is writable
[ -d /etc/udev/rules.d ] || \
eudev_dir_writeable /etc/udev && \
mkdir -p /etc/udev/rules.d

# only continue if rules-directory is writable
eudev_dir_writeable /etc/udev/rules.d || return 0

local file dest
for file in /run/udev/tmp-rules--*; do
dest=${file##*tmp-rules--}
[ "$dest" = '*' ] && break
type=${dest##70-persistent-}
type=${type%%.rules}
cat "$file" >> /etc/udev/rules.d/"$dest" && rm -f "$file"
done
}

eudev_start()
{
eudev_start_pre
udevd -d
# store persistent-rules that got created while booting
# when / was still read-only
eudev_store_persistent_rules
# Populating /dev with existing devices through uevents"
udevadm trigger --type=subsystems --action=add
udevadm trigger --type=devices --action=add
}
</nowiki>}}
After the following section, which populates {{ic|/dev}}
{{Text art|<nowiki>
mount -t devtmpfs -o exec,nosuid,mode=0755,size=2M devtmpfs /dev 2>/dev/null \
|| mount -t tmpfs -o exec,nosuid,mode=0755,size=2M tmpfs /dev

# pty device nodes (later system will need it)
[ -c /dev/ptmx ] || mknod -m 666 /dev/ptmx c 5 2
[ -d /dev/pts ] || mkdir -m 755 /dev/pts
mount -t devpts -o gid=5,mode=0620,noexec,nosuid devpts /dev/pts

# shared memory area (later system will need it)
[ -d /dev/shm ] || mkdir /dev/shm
mount -t tmpfs -o nodev,nosuid,noexec shm /dev/shm
</nowiki>}}
add this paragraph.
{{Text art|<nowiki>
# persistent device names from eudev
if [ -f /sbin/udevadm ]; then
eudev_start
fi
</nowiki>}}
To let {{ic|zpool import}} to use persistent names, use zpool.cache to import pool
zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID
echo /etc/zfs/zpool.cache >> /etc/mkinitfs/features.d/zfs.files
replace {{ic|zpool import}} with {{ic|zpool import -c /etc/zfs/zpool.cache}}
sed -i 's|-d /dev|-c /etc/zfs/zpool.cache|g' /usr/share/mkinitfs/initramfs-init

= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-31T03:34:44Z

R2: /* Enable persistent device names in initramfs */ zpool import fix

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.

{{ic|shadow}} is available in community repo. Enable it first:
vi /etc/apk/repositories
# uncomment community line
Install
apk add shadow sudo eudev

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/sh -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =
As of this writing, the initramfs has a bug for entering ZFS password at boot. When booting the system, root dataset will fail to mount {{ic|sh: `active`, unknown operand}}and drop into emergency shell.

In {{ic|/usr/share/mkinitfs/initramfs-init}}:
# Ask for encryption password
if [ $(zpool list -H -o feature@encryption $_root_pool) = "active" ]; then
# replace underline with true will fix it
# if true; then
local _encryption_root=$(zfs get -H -o value encryptionroot $_root_vol)
if [ "$_encryption_root" != "-" ]; then
eval zfs load-key $_encryption_root
fi
fi

Until this is fixed, we need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs rpool_$UUID/ROOT/default /sysroot
= Enable persistent device names in initramfs =
Special modifications need to be made to populate {{ic|/dev/disk/by-*}} in initramfs.

Ensure {{ic|eudev}} is installed
apk add eudev
Create {{ic|/etc/mkinitfs/features.d/eudev.files}} to add {{ic|eudev}} to initramfs.
tee /etc/mkinitfs/features.d/eudev.files << EOF
/bin/udevadm
/sbin/udevadm
/sbin/udevd
/etc/udev/*
/lib/udev/*
/usr/lib/libudev*
EOF
Edit {{ic|/usr/share/mkinitfs/initramfs-init}}.

Add functions from {{ic|/etc/init.d/udev*}} at the beginning of the file.
{{Text art|<nowiki>
# persistent device names from eudev, see /etc/init.d/udev*
eudev_start_pre() {
# load unix domain sockets if built as module, Bug #221253
# and not yet loaded, Bug #363549
if [ ! -e /proc/net/unix ]; then
if ! modprobe unix; then
eerror "Cannot load the unix domain socket module"
return 1
fi
fi

if [ -e /proc/sys/kernel/hotplug ]; then
echo "" >/proc/sys/kernel/hotplug
fi

return 0
}

eudev_dir_writeable()
{
touch "$1"/.test.$$ 2>/dev/null && rm "$1"/.test.$$
}

# store persistent-rules that got created while booting
# when / was still read-only
eudev_store_persistent_rules()
{
# create /etc/udev/rules.d if it does not exist and /etc/udev is writable
[ -d /etc/udev/rules.d ] || \
eudev_dir_writeable /etc/udev && \
mkdir -p /etc/udev/rules.d

# only continue if rules-directory is writable
eudev_dir_writeable /etc/udev/rules.d || return 0

local file dest
for file in /run/udev/tmp-rules--*; do
dest=${file##*tmp-rules--}
[ "$dest" = '*' ] && break
type=${dest##70-persistent-}
type=${type%%.rules}
cat "$file" >> /etc/udev/rules.d/"$dest" && rm -f "$file"
done
}

eudev_start()
{
eudev_start_pre
udevd -d
# store persistent-rules that got created while booting
# when / was still read-only
eudev_store_persistent_rules
# Populating /dev with existing devices through uevents"
udevadm trigger --type=subsystems --action=add
udevadm trigger --type=devices --action=add
}
</nowiki>}}
After the following section, which populates {{ic|/dev}}
{{Text art|<nowiki>
mount -t devtmpfs -o exec,nosuid,mode=0755,size=2M devtmpfs /dev 2>/dev/null \
|| mount -t tmpfs -o exec,nosuid,mode=0755,size=2M tmpfs /dev

# pty device nodes (later system will need it)
[ -c /dev/ptmx ] || mknod -m 666 /dev/ptmx c 5 2
[ -d /dev/pts ] || mkdir -m 755 /dev/pts
mount -t devpts -o gid=5,mode=0620,noexec,nosuid devpts /dev/pts

# shared memory area (later system will need it)
[ -d /dev/shm ] || mkdir /dev/shm
mount -t tmpfs -o nodev,nosuid,noexec shm /dev/shm
</nowiki>}}
add this paragraph.
{{Text art|<nowiki>
# persistent device names from eudev
if [ -f /sbin/udevadm ]; then
eudev_start
fi
</nowiki>}}
To let {{ic|zpool import}} to use persistent names, replace {{ic|/dev}} with {{ic|/dev/disk-id}}
sed -i 's|zpool import -N -d /dev|zpool import -N -d /dev/disk/by-id|g' /usr/share/mkinitfs/initramfs-init

= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-31T03:29:04Z

R2: /* Enable persistent device names in initramfs */ update

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.

{{ic|shadow}} is available in community repo. Enable it first:
vi /etc/apk/repositories
# uncomment community line
Install
apk add shadow sudo eudev

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/sh -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =
As of this writing, the initramfs has a bug for entering ZFS password at boot. When booting the system, root dataset will fail to mount {{ic|sh: `active`, unknown operand}}and drop into emergency shell.

In {{ic|/usr/share/mkinitfs/initramfs-init}}:
# Ask for encryption password
if [ $(zpool list -H -o feature@encryption $_root_pool) = "active" ]; then
# replace underline with true will fix it
# if true; then
local _encryption_root=$(zfs get -H -o value encryptionroot $_root_vol)
if [ "$_encryption_root" != "-" ]; then
eval zfs load-key $_encryption_root
fi
fi

Until this is fixed, we need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs rpool_$UUID/ROOT/default /sysroot
= Enable persistent device names in initramfs =
Special modifications need to be made to populate {{ic|/dev/disk/by-*}} in initramfs.

Ensure {{ic|eudev}} is installed
apk add eudev
Create {{ic|/etc/mkinitfs/features.d/eudev.files}} to add {{ic|eudev}} to initramfs.
tee /etc/mkinitfs/features.d/eudev.files << EOF
/bin/udevadm
/sbin/udevadm
/sbin/udevd
/etc/udev/*
/lib/udev/*
/usr/lib/libudev*
EOF
Edit {{ic|/usr/share/mkinitfs/initramfs-init}}.
Add functions from {{ic|/etc/init.d/udev*}} at the beginning of the file.
{{Text art|<nowiki>
# persistent device names from eudev, see /etc/init.d/udev*
eudev_start_pre() {
# load unix domain sockets if built as module, Bug #221253
# and not yet loaded, Bug #363549
if [ ! -e /proc/net/unix ]; then
if ! modprobe unix; then
eerror "Cannot load the unix domain socket module"
return 1
fi
fi

if [ -e /proc/sys/kernel/hotplug ]; then
echo "" >/proc/sys/kernel/hotplug
fi

return 0
}

eudev_dir_writeable()
{
touch "$1"/.test.$$ 2>/dev/null && rm "$1"/.test.$$
}

# store persistent-rules that got created while booting
# when / was still read-only
eudev_store_persistent_rules()
{
# create /etc/udev/rules.d if it does not exist and /etc/udev is writable
[ -d /etc/udev/rules.d ] || \
eudev_dir_writeable /etc/udev && \
mkdir -p /etc/udev/rules.d

# only continue if rules-directory is writable
eudev_dir_writeable /etc/udev/rules.d || return 0

local file dest
for file in /run/udev/tmp-rules--*; do
dest=${file##*tmp-rules--}
[ "$dest" = '*' ] && break
type=${dest##70-persistent-}
type=${type%%.rules}
cat "$file" >> /etc/udev/rules.d/"$dest" && rm -f "$file"
done
}

eudev_start()
{
eudev_start_pre
udevd -d
# store persistent-rules that got created while booting
# when / was still read-only
eudev_store_persistent_rules
# Populating /dev with existing devices through uevents"
udevadm trigger --type=subsystems --action=add
udevadm trigger --type=devices --action=add
}
</nowiki>}}
After the following section, which populates {{ic|/dev}}
{{Text art|<nowiki>
mount -t devtmpfs -o exec,nosuid,mode=0755,size=2M devtmpfs /dev 2>/dev/null \
|| mount -t tmpfs -o exec,nosuid,mode=0755,size=2M tmpfs /dev

# pty device nodes (later system will need it)
[ -c /dev/ptmx ] || mknod -m 666 /dev/ptmx c 5 2
[ -d /dev/pts ] || mkdir -m 755 /dev/pts
mount -t devpts -o gid=5,mode=0620,noexec,nosuid devpts /dev/pts

# shared memory area (later system will need it)
[ -d /dev/shm ] || mkdir /dev/shm
mount -t tmpfs -o nodev,nosuid,noexec shm /dev/shm
</nowiki>}}
add this paragraph.
{{Text art|<nowiki>
# persistent device names from eudev
if [ -f /sbin/udevadm ]; then
eudev_start
fi
</nowiki>}}

= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-31T02:40:44Z

R2: /* Persistent device names not available in initramfs */ update

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.

{{ic|shadow}} is available in community repo. Enable it first:
vi /etc/apk/repositories
# uncomment community line
Install
apk add shadow sudo eudev

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/sh -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =
As of this writing, the initramfs has a bug for entering ZFS password at boot. When booting the system, root dataset will fail to mount {{ic|sh: `active`, unknown operand}}and drop into emergency shell.

In {{ic|/usr/share/mkinitfs/initramfs-init}}:
# Ask for encryption password
if [ $(zpool list -H -o feature@encryption $_root_pool) = "active" ]; then
# replace underline with true will fix it
# if true; then
local _encryption_root=$(zfs get -H -o value encryptionroot $_root_vol)
if [ "$_encryption_root" != "-" ]; then
eval zfs load-key $_encryption_root
fi
fi

Until this is fixed, we need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs rpool_$UUID/ROOT/default /sysroot
= Enable persistent device names in initramfs =
Special modifications need to be made to populate {{ic|/dev/disk/by-*}} in initramfs.

Ensure {{ic|eudev}} is installed
apk add eudev
Create {{ic|/etc/mkinitfs/features.d/eudev.files}} to add {{ic|eudev}} to initramfs.
tee /etc/mkinitfs/features.d/eudev.files << EOF
/bin/udevadm
/sbin/udevadm
/sbin/udevd
/etc/udev/*
/lib/udev/*
/usr/lib/libudev*
EOF
Edit {{ic|/usr/share/mkinitfs/initramfs-init}}. After the following section, which populates {{ic|/dev}}
{{Text art|<nowiki>
mount -t devtmpfs -o exec,nosuid,mode=0755,size=2M devtmpfs /dev 2>/dev/null \
|| mount -t tmpfs -o exec,nosuid,mode=0755,size=2M tmpfs /dev

# pty device nodes (later system will need it)
[ -c /dev/ptmx ] || mknod -m 666 /dev/ptmx c 5 2
[ -d /dev/pts ] || mkdir -m 755 /dev/pts
mount -t devpts -o gid=5,mode=0620,noexec,nosuid devpts /dev/pts

# shared memory area (later system will need it)
[ -d /dev/shm ] || mkdir /dev/shm
mount -t tmpfs -o nodev,nosuid,noexec shm /dev/shm
</nowiki>}}
add this paragraph.
{{Text art|<nowiki>
# persistent device names from eudev
if [ -f /sbin/udevadm ]; then
udevadm trigger --type=subsystems --action=add
udevadm trigger --type=devices --action=add
fi
</nowiki>}}

= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-31T01:01:17Z

R2: /* Persistent device names not available in initramfs */ grammar

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.

{{ic|shadow}} is available in community repo. Enable it first:
vi /etc/apk/repositories
# uncomment community line
Install
apk add shadow sudo eudev

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/sh -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =
As of this writing, the initramfs has a bug for entering ZFS password at boot. When booting the system, root dataset will fail to mount {{ic|sh: `active`, unknown operand}}and drop into emergency shell.

In {{ic|/usr/share/mkinitfs/initramfs-init}}:
# Ask for encryption password
if [ $(zpool list -H -o feature@encryption $_root_pool) = "active" ]; then
# replace underline with true will fix it
# if true; then
local _encryption_root=$(zfs get -H -o value encryptionroot $_root_vol)
if [ "$_encryption_root" != "-" ]; then
eval zfs load-key $_encryption_root
fi
fi

Until this is fixed, we need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs rpool_$UUID/ROOT/default /sysroot
= Persistent device names not available in initramfs =
Currently, random block device names such as {{ic|/dev/sda}} is still used in initramfs, {{ic|/dev/disk/by-*}} are not populated until entering system.

Still need to find a way to add eudev to initramfs.

= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-31T01:00:44Z

R2: /* Reboot */ missing eudev to initramfs

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.

{{ic|shadow}} is available in community repo. Enable it first:
vi /etc/apk/repositories
# uncomment community line
Install
apk add shadow sudo eudev

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/sh -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =
As of this writing, the initramfs has a bug for entering ZFS password at boot. When booting the system, root dataset will fail to mount {{ic|sh: `active`, unknown operand}}and drop into emergency shell.

In {{ic|/usr/share/mkinitfs/initramfs-init}}:
# Ask for encryption password
if [ $(zpool list -H -o feature@encryption $_root_pool) = "active" ]; then
# replace underline with true will fix it
# if true; then
local _encryption_root=$(zfs get -H -o value encryptionroot $_root_vol)
if [ "$_encryption_root" != "-" ]; then
eval zfs load-key $_encryption_root
fi
fi

Until this is fixed, we need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs rpool_$UUID/ROOT/default /sysroot
= Persistent device names not available in initramfs =
Currently, random block device names such as {{ic|/dev/sda}} is still used in initramfs, {{ic|/dev/disk/by-*}} are not populated until entering system.

Still finding a way to add eudev to initramfs.

= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-31T00:57:46Z

R2: /* Enable ZFS services */ handled by fstab, not used

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.

{{ic|shadow}} is available in community repo. Enable it first:
vi /etc/apk/repositories
# uncomment community line
Install
apk add shadow sudo eudev

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/sh -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =
As of this writing, the initramfs has a bug for entering ZFS password at boot. When booting the system, root dataset will fail to mount {{ic|sh: `active`, unknown operand}}and drop into emergency shell.

In {{ic|/usr/share/mkinitfs/initramfs-init}}:
# Ask for encryption password
if [ $(zpool list -H -o feature@encryption $_root_pool) = "active" ]; then
# replace underline with true will fix it
# if true; then
local _encryption_root=$(zfs get -H -o value encryptionroot $_root_vol)
if [ "$_encryption_root" != "-" ]; then
eval zfs load-key $_encryption_root
fi
fi

Until this is fixed, we need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs rpool_$UUID/ROOT/default /sysroot

= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-31T00:50:49Z

R2: /* Reboot */ fix

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.

{{ic|shadow}} is available in community repo. Enable it first:
vi /etc/apk/repositories
# uncomment community line
Install
apk add shadow sudo eudev

= Enable ZFS services =
rc-update add zfs-import sysinit
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit
rc-update add udev-trigger sysinit

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/sh -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =
As of this writing, the initramfs has a bug for entering ZFS password at boot. When booting the system, root dataset will fail to mount {{ic|sh: `active`, unknown operand}}and drop into emergency shell.

In {{ic|/usr/share/mkinitfs/initramfs-init}}:
# Ask for encryption password
if [ $(zpool list -H -o feature@encryption $_root_pool) = "active" ]; then
# replace underline with true will fix it
# if true; then
local _encryption_root=$(zfs get -H -o value encryptionroot $_root_vol)
if [ "$_encryption_root" != "-" ]; then
eval zfs load-key $_encryption_root
fi
fi

Until this is fixed, we need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs rpool_$UUID/ROOT/default /sysroot

= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-31T00:43:07Z

R2: /* Reboot */ fixes and bugs

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.

{{ic|shadow}} is available in community repo. Enable it first:
vi /etc/apk/repositories
# uncomment community line
Install
apk add shadow sudo eudev

= Enable ZFS services =
rc-update add zfs-import sysinit
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit
rc-update add udev-trigger sysinit

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/sh -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =
As of this writing, the initramfs has a bug for entering ZFS password at boot. When booting the system, root dataset will fail to mount {{ic|sh: `active`, unknown operand}}and drop into emergency shell.

In {{ic|/usr/share/mkinitfs/initramfs-init}}:
# Ask for encryption password
if [ $(zpool list -H -o feature@encryption $_root_pool) = "active" ]; then
local _encryption_root=$(zfs get -H -o value encryptionroot $_root_vol)
if [ "$_encryption_root" != "-" ]; then
eval zfs load-key $_encryption_root
fi
fi

Until this is fixed, we need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs rpool_$UUID/ROOT/default /sysroot

= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-30T17:25:36Z

R2: /* Optional: Enable encrypted swap partition */ mkinitfs

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.

{{ic|shadow}} is available in community repo. Enable it first:
vi /etc/apk/repositories
# uncomment community line
Install
apk add shadow sudo eudev

= Enable ZFS services =
rc-update add zfs-import sysinit
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit
rc-update add udev-trigger sysinit

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/sh -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs with {{ic|mkinitfs}}.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =
As of this writing, the initramfs lacks support for entering ZFS password at boot. When booting the system, root dataset will simply fail to mount and drop into emergency shell.

We need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs $(cat /proc/cmdline | sed 's|.*ZFS=||' | awk '{ print $1 }') /sysroot
ArchZFS project solved this with a sh script, available [https://github.com/archzfs/archzfs/blob/master/src/zfs-utils/zfs-utils.initcpio.hook here].

Note that initramfs will attempt to import all visible pools on boot, even exported ones. Be sure only pools you want are connected. See {{ic|/etc/init.d/zfs-import}}.

= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-30T17:17:39Z

R2: /* Reboot */ note for initramfs

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.

{{ic|shadow}} is available in community repo. Enable it first:
vi /etc/apk/repositories
# uncomment community line
Install
apk add shadow sudo eudev

= Enable ZFS services =
rc-update add zfs-import sysinit
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit
rc-update add udev-trigger sysinit

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/sh -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =
As of this writing, the initramfs lacks support for entering ZFS password at boot. When booting the system, root dataset will simply fail to mount and drop into emergency shell.

We need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs $(cat /proc/cmdline | sed 's|.*ZFS=||' | awk '{ print $1 }') /sysroot
ArchZFS project solved this with a sh script, available [https://github.com/archzfs/archzfs/blob/master/src/zfs-utils/zfs-utils.initcpio.hook here].

Note that initramfs will attempt to import all visible pools on boot, even exported ones. Be sure only pools you want are connected. See {{ic|/etc/init.d/zfs-import}}.

= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-30T16:59:50Z

R2: /* Reboot */ disk space stat

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.

{{ic|shadow}} is available in community repo. Enable it first:
vi /etc/apk/repositories
# uncomment community line
Install
apk add shadow sudo eudev

= Enable ZFS services =
rc-update add zfs-import sysinit
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit
rc-update add udev-trigger sysinit

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/sh -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =
As of this writing, the initramfs lacks support for entering ZFS password at boot. When booting the system, root dataset will simply fail to mount and drop into emergency shell.

We need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs $(cat /proc/cmdline | sed 's|.*ZFS=||' | awk '{ print $1 }') /sysroot
ArchZFS project solved this with a sh script, available [https://github.com/archzfs/archzfs/blob/master/src/zfs-utils/zfs-utils.initcpio.hook here].

= Disk space stat =
Without optional swap or cryptsetup:
*bpool used 25.2M
*rpool used 491M
*efi used 416K

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-30T16:56:18Z

R2: /* Reboot */ use command

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.

{{ic|shadow}} is available in community repo. Enable it first:
vi /etc/apk/repositories
# uncomment community line
Install
apk add shadow sudo eudev

= Enable ZFS services =
rc-update add zfs-import sysinit
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit
rc-update add udev-trigger sysinit

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/sh -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =
As of this writing, the initramfs lacks support for entering ZFS password at boot. When booting the system, root dataset will simply fail to mount and drop into emergency shell.

We need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs $(cat /proc/cmdline | sed 's|.*ZFS=||' | awk '{ print $1 }') /sysroot
ArchZFS project solved this with a sh script, available [https://github.com/archzfs/archzfs/blob/master/src/zfs-utils/zfs-utils.initcpio.hook here].

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-30T16:49:51Z

R2: /* Enable encrypted swap partition */ optional

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.

{{ic|shadow}} is available in community repo. Enable it first:
vi /etc/apk/repositories
# uncomment community line
Install
apk add shadow sudo eudev

= Enable ZFS services =
rc-update add zfs-import sysinit
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit
rc-update add udev-trigger sysinit

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/sh -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Optional: Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =
As of this writing, the initramfs lacks support for entering ZFS password at boot. When booting the system, root dataset will simply fail to mount and drop into emergency shell.

We need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs rpool_$poolUUID/ROOT/default /sysroot
ArchZFS project solved this with a sh script, available [https://github.com/archzfs/archzfs/blob/master/src/zfs-utils/zfs-utils.initcpio.hook here].

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-30T16:49:21Z

R2: /* Add normal user account */ use ash

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.

{{ic|shadow}} is available in community repo. Enable it first:
vi /etc/apk/repositories
# uncomment community line
Install
apk add shadow sudo eudev

= Enable ZFS services =
rc-update add zfs-import sysinit
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit
rc-update add udev-trigger sysinit

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/sh -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =
As of this writing, the initramfs lacks support for entering ZFS password at boot. When booting the system, root dataset will simply fail to mount and drop into emergency shell.

We need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs rpool_$poolUUID/ROOT/default /sysroot
ArchZFS project solved this with a sh script, available [https://github.com/archzfs/archzfs/blob/master/src/zfs-utils/zfs-utils.initcpio.hook here].

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-30T16:48:23Z

R2: /* Install packages */ enable community

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.

{{ic|shadow}} is available in community repo. Enable it first:
vi /etc/apk/repositories
# uncomment community line
Install
apk add shadow sudo eudev

= Enable ZFS services =
rc-update add zfs-import sysinit
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit
rc-update add udev-trigger sysinit

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/bash -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd
= Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =
As of this writing, the initramfs lacks support for entering ZFS password at boot. When booting the system, root dataset will simply fail to mount and drop into emergency shell.

We need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs rpool_$poolUUID/ROOT/default /sysroot
ArchZFS project solved this with a sh script, available [https://github.com/archzfs/archzfs/blob/master/src/zfs-utils/zfs-utils.initcpio.hook here].

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-30T16:43:37Z

R2: /* Enable encrypted swap partition */ rebuild

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.
apk add shadow sudo eudev

= Enable ZFS services =
rc-update add zfs-import sysinit
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit
rc-update add udev-trigger sysinit

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/bash -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd
= Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab
Rebuild initramfs.

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =
As of this writing, the initramfs lacks support for entering ZFS password at boot. When booting the system, root dataset will simply fail to mount and drop into emergency shell.

We need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs rpool_$poolUUID/ROOT/default /sysroot
ArchZFS project solved this with a sh script, available [https://github.com/archzfs/archzfs/blob/master/src/zfs-utils/zfs-utils.initcpio.hook here].

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-30T16:41:16Z

R2: /* Format and mount EFI partition */ fs type

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.
apk add shadow sudo eudev

= Enable ZFS services =
rc-update add zfs-import sysinit
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit
rc-update add udev-trigger sysinit

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/bash -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd
= Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =
As of this writing, the initramfs lacks support for entering ZFS password at boot. When booting the system, root dataset will simply fail to mount and drop into emergency shell.

We need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs rpool_$poolUUID/ROOT/default /sysroot
ArchZFS project solved this with a sh script, available [https://github.com/archzfs/archzfs/blob/master/src/zfs-utils/zfs-utils.initcpio.hook here].

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-30T16:39:24Z

R2: /* Variables */ 8 char min

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.
apk add shadow sudo eudev

= Enable ZFS services =
rc-update add zfs-import sysinit
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit
rc-update add udev-trigger sysinit

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/bash -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd
= Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =
As of this writing, the initramfs lacks support for entering ZFS password at boot. When booting the system, root dataset will simply fail to mount and drop into emergency shell.

We need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs rpool_$poolUUID/ROOT/default /sysroot
ArchZFS project solved this with a sh script, available [https://github.com/archzfs/archzfs/blob/master/src/zfs-utils/zfs-utils.initcpio.hook here].

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-30T16:16:22Z

R2: /* Add normal user account */ swap instructions

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.
apk add shadow sudo eudev

= Enable ZFS services =
rc-update add zfs-import sysinit
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit
rc-update add udev-trigger sysinit

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/bash -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd
= Enable encrypted swap partition =
Install {{ic|cryptsetup}}
apk add cryptsetup
Edit the <code>/mnt/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the <code>features</code> parameter:
features="ata base ide scsi usb virtio ext4 lvm cryptsetup zfs"
Add relevant lines in {{ic|fstab}} and {{ic|crypttab}}. Replace {{ic|$DISK}} with actual disk.
echo swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =
As of this writing, the initramfs lacks support for entering ZFS password at boot. When booting the system, root dataset will simply fail to mount and drop into emergency shell.

We need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs rpool_$poolUUID/ROOT/default /sysroot
ArchZFS project solved this with a sh script, available [https://github.com/archzfs/archzfs/blob/master/src/zfs-utils/zfs-utils.initcpio.hook here].

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-30T16:05:44Z

R2: /* Recovery in Live environment */ remove reference to archlinux

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.
apk add shadow sudo eudev

= Enable ZFS services =
rc-update add zfs-import sysinit
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit
rc-update add udev-trigger sysinit

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/bash -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =
As of this writing, the initramfs lacks support for entering ZFS password at boot. When booting the system, root dataset will simply fail to mount and drop into emergency shell.

We need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs rpool_$poolUUID/ROOT/default /sysroot
ArchZFS project solved this with a sh script, available [https://github.com/archzfs/archzfs/blob/master/src/zfs-utils/zfs-utils.initcpio.hook here].

= Recovery in Live environment =
After installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh

After chroot, mount {{ic|/efi}}
mount /boot/efi
After fixing the system, don't forget to umount and export the pools:
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-30T16:03:35Z

R2: /* Reboot */ recovery

= Objectives =
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.

Except EFI system partition and boot pool {{ic|/boot}}, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.

To do an unencrypted setup, simply omit {{ic|-O keylocation -O keyformat}} when creating root pool.

= Notes =
== Swap on ZFS will cause dead lock ==
You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure. See [https://github.com/openzfs/zfs/issues/7734] This guide will set up swap on a separate partition with plain dm-crypt.

Resume from swap is not possible, because the key of swap partition can not be stored in the unencrypted boot pool. Busybox initramfs only supports unlocking exactly one LUKS container at boot, therefore boot pool and swap partition can not be both LUKS encrypted. A possible workaround is to import and mount boot pool after booting the system via systemd service.

== Resume from ZFS will corrupt the pool ==
ZFS does not support freeze/thaw operations, which is required for resuming from hibernation, or suspend to disk. Attempt to resume from a swap on ZFS '''WILL''' corrupt the pool. See [https://github.com/openzfs/zfs/issues/260]
== Encrypted boot pool ==
GRUB supports booting from LUKS-1 encrypted containers. Therefore, it is possible to encrypt both boot pool and root pool to achieve full disk encryption.

To do this, format boot pool partition as LUKS-1 container and supply the encryption password here. Use keyfile for root pool and embed the keyfile in initramfs.

Since there isn't any sensitive information in {{ic|/boot}} anyway (unless you want to use a persistent LUKS encrypted swap partition for resume from hibernation), encrypting boot pool provides no meaningful benefit and complicates the installation and recovery process.

= Pre-installation =
UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.

'''Existing data on target disk(s) will be destroyed.'''

Download the '''extended''' release from https://www.alpinelinux.org/downloads/, it's shipped with ZFS kernel module.

Write it to a USB and boot from it.

== Setup live environment ==
Run the following command to setup the live environment, select disk=none at the last step when asked for disk mode. See [[Installation#Questions_asked_by_setup-alpine]].
setup-alpine
The settings given here will be copied to the target system later by {{ic|setup-disk}}.

== Install system utilities ==
apk update
apk add eudev sgdisk grub-efi zfs
modprobe zfs
Here we must install eudev to have persistent block device names. '''Do not use''' /dev/sda for ZFS pools.
rc-update add udev-trigger sysinit
/etc/init.d/udev-trigger start

= Variables =
In this step, we will set some variables to make our installation process easier.
DISK=/dev/disk/by-id/ata-HXY_120G_YS
Use unique disk path instead of {{ic|/dev/sda}} to ensure the correct partition can be found by ZFS.

Other variables
TARGET_USERNAME='your username'
ENCRYPTION_PWD='your root pool encryption password'
TARGET_USERPWD='user account password'
Create a mountpoint
MOUNTPOINT=`mktemp -d`
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

= Partitioning =
For a single disk, UEFI installation, we need to create at lease 3 partitions:
* EFI system partition
* Boot pool partition
* Root pool partition
Since [[GRUB]] only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.

Clear the partition table on the target disk and create EFI, boot and root pool parititions:
sgdisk --zap-all $DISK
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK # boot pool
sgdisk -n3:0:0 $DISK # root pool
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.

== Optional: Swap partition ==
[[Swap]] support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)

If you want to use swap, reserve some space at the end of disk when creating root pool:
sgdisk -n3:0:-8G $DISK # root pool, reserve 8GB for swap at the end of the disk
sgdisk -n4:0:0 $DISK # swap partition

= Boot and root pool creation =
As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no {{ic|feature@}} is supplied.

Here we explicitly enable those GRUB can support.
zpool create \
-o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/boot -R $MOUNTPOINT \
bpool_$poolUUID $DISK-part2
Nothing is stored directly under bpool and rpool, hence {{ic|1=canmount=off}}. The respective {{ic|mountpoint}} properties are more symbolic than practical.

For root pool all available features are enabled by default
echo $ENCRYPTION_PWD | zpool create \
-o ashift=12 \
-O encryption=aes-256-gcm \
-O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
rpool_$poolUUID $DISK-part3

== Notes for multi-disk ==
For mirror:
zpool create \
... \
bpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part2 \
/dev/disk/by-id/target_disk2-part2
zpool create \
... \
rpool_$poolUUID mirror \
/dev/disk/by-id/target_disk1-part3 \
/dev/disk/by-id/target_disk2-part3
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.

= Dataset creation =
This layout is intended to separate root file system from persistent files. See [https://wiki.archlinux.org/index.php/User:M0p/Root_on_ZFS_Native_Encryption/Layout] for a description.
{{Text art|<nowiki>
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
zfs mount bpool_$poolUUID/BOOT/default
d='usr var var/lib'
for i in $d; do zfs create -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME
</nowiki>}}

= Format and mount EFI partition =
mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount $DISK-part1 $MOUNTPOINT/boot/efi

= Install Alpine Linux to target disk =
== Preparations ==
GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=YES.
export ZPOOL_VDEV_NAME_PATH=YES
setup-disk refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk

== Run setup-disk ==
BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT
Note that grub-probe will still fail despite ZPOOL_VDEV_NAME_PATH=YES variable set above. We will deal with this later inside chroot.

= Chroot into new system =
mount --rbind /dev $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys $MOUNTPOINT/sys
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh

= Finish GRUB installation =
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.

Apply GRUB ZFS fix:
export ZPOOL_VDEV_NAME_PATH=YES
Apply fixes in WARNING.
== WARNING ==
1. GRUB will fail to detect the ZFS filesystem of {{ic|/boot}} with {{ic|stat}} from Busybox.

See [https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkconfig.in source file of grub-mkconfig], the problem is:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# will fail with `grub-probe: error: unknown filesystem.`
GRUB_FS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2> /dev/null || echo unknown`"
# will also fail. The final fall back is
if [ x"$GRUB_FS" = xunknown ]; then
GRUB_FS="$(stat -f -c %T / || echo unknown)"
fi
# `stat` from coreutils will return `zfs`, the correct answer
# `stat` from busybox will return `UNKNOWN`, cause `10_linux` script to fail
Therefore we need to install {{ic|coreutils}}.
apk add coreutils

2. GRUB will stuff an empty result if it does not support root pool.

GRUB is lagging behind development of OpenZFS, see [https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00239.html]. A temporary fix is to replace detection of rpool with the method given in patch.
sed -i "s/rpool=.*/rpool=\`zdb -l \${GRUB_DEVICE} \| awk -F \\\' '\/ name\/ { print \$2 }'\`/" /etc/grub.d/10_linux
Need to be applied upon every GRUB update until the patch is merged.

== Generate grub.cfg ==
After applying fixes, finally run
grub-mkconfig -o /boot/grub/grub.cfg

= Install packages =
These packages are used for creating a common user account. Root account is accessed with sudo. Also package for persisitent block device name must be installed.
apk add shadow sudo eudev

= Enable ZFS services =
rc-update add zfs-import sysinit
rc-update add zfs-mount sysinit
rc-update add zfs-zed sysinit
rc-update add udev-trigger sysinit

= Enable sudo access for wheel group =
mv /etc/sudoers /etc/sudoers.original
tee /etc/sudoers << EOF
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
EOF

= Add normal user account =
useradd -s /bin/bash -U -G wheel,video -d /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME:$TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd

= Finish installation =
Take a snapshot for the clean installation for future use and export all pools.
exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install
Pools must be exported before reboot, or they will fail to be imported on boot.
mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

= Reboot =
As of this writing, the initramfs lacks support for entering ZFS password at boot. When booting the system, root dataset will simply fail to mount and drop into emergency shell.

We need to manually load the key and mount root dataset with
zfs load-key -a
# enter password
mount -t zfs rpool_$poolUUID/ROOT/default /sysroot
ArchZFS project solved this with a sh script, available [https://github.com/archzfs/archzfs/blob/master/src/zfs-utils/zfs-utils.initcpio.hook here].

= Recovery in Live environment =
After adding archzfs repo and installing zfs packages, run the following command:

Create a mount point and store encryption password in a variable:
MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'
Find the unique UUID of your pool with
zpool import
Import rpool without mounting datasets: {{ic|-N}} for not mounting all datasets; {{ic|-R}} for alternate root.
poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID
Load encryption key
echo $ENCRYPTION_PWD | zfs load-key -a
As {{ic|1=canmount=noauto}} is set for {{ic|/}} dataset, we have to mount it manually. To find the dataset, use
zfs list rpool_$poolUUID/ROOT
Mount {{ic|/}} dataset
zfs mount rpool_$UUID/ROOT/''$dataset''
Mount other datasets
zfs mount -a
Import bpool
zpool import -N -R $MOUNTPOINT bpool_$UUID
Find and mount the {{ic|/boot}} dataset, same as above.
zfs list bpool_$UUID/BOOT
zfs mount bpool_$UUID/BOOT/''$dataset''
Chroot
arch-chroot $MOUNTPOINT bash --login
After chroot, mount {{ic|/efi}}
mount /efi
After fixing the system, don't forget to umount and export the pools:
umount $MOUNTPOINT/efi
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID

Root on ZFS with native encryption

2020-12-30T15:56:28Z

R2: /* Finish GRUB installation */ fix order

Root on ZFS with native encryption

2020-12-30T15:48:05Z

R2: /* WARNING */ missing file name

Root on ZFS with native encryption

2020-12-30T15:47:01Z

R2: /* WARNING */ fix format

Root on ZFS with native encryption

2020-12-30T15:46:06Z

R2: /* WARNING */ update

Root on ZFS with native encryption

2020-12-30T15:25:27Z

R2: /* Finish GRUB installation */ warning

Root on ZFS with native encryption

2020-12-30T14:09:40Z

R2: /* Finish GRUB installation */ warning

Root on ZFS with native encryption

2020-12-30T13:35:55Z

R2: /* Dataset creation */ not needed

Root on ZFS with native encryption

2020-12-30T13:11:03Z

R2: /* Dataset creation */ note for legacy mountpoints

Root on ZFS with native encryption

2020-12-30T13:00:35Z

R2: /* Preparations */ missing subtitle

Root on ZFS with native encryption

2020-12-30T12:45:28Z

R2: /* Dataset creation */ details for dataset layout

Root on ZFS with native encryption

2020-12-30T12:40:54Z

R2: /* Setup live environment */ inline code

Root on ZFS with native encryption

2020-12-30T12:40:04Z

R2: /* = Preparations */ fix title

Template:Text art

2020-12-30T12:38:46Z

R2: copy from archwiki

<noinclude>
{{Template}}

Block code that does not wrap on narrow screens, intended for [[Wikipedia:Text art|Text art]].

Use [[Template:bc]] for regular block codes.

== Usage ==

{{bc|<nowiki>{{Text art|<nowiki>
Alpine Linux
</nowiki>}}</nowiki>}}

== Example ==

{{Text art|<nowiki>
Alpine Linux
</nowiki>}}
</noinclude><includeonly><pre style="white-space: pre !important; font-variant-ligatures: no-common-ligatures;">{{{code|{{{1|{{META Error}}}}}}}}</pre></includeonly>

Template:Ic

2020-12-30T12:36:59Z

R2: inline code

<noinclude>{{Template}} {{DISPLAYTITLE:Template:ic}}

Inline code.

* Use [[Template:bc]] for block code without header.
* Use [[Template:hc]] for block code with header.

==Usage==

{{bc|<nowiki>{{ic|code}}</nowiki>}}

{{Tip|When representing keyboard keys, you can use [[Wikipedia:List of XML and HTML character entity references|HTML entities]] {{ic|&uarr;}}, {{ic|&rarr;}}, {{ic|&darr;}} and {{ic|&larr;}} to depict arrow keys: {{ic|↑}}, {{ic|→}}, {{ic|↓}}, {{ic|←}}}}

==Example==

{{ic|code}}</noinclude><includeonly><code>{{{code|{{{1|{{META Error}}}}}}}}</code></includeonly>

Root on ZFS with native encryption

2020-12-30T12:35:38Z

R2: update