Root on ZFS with native encryption: Difference between revisions

From Alpine Linux
(rm)
(del)
Line 1: Line 1:
This guide aims to setup encrypted Alpine Linux on ZFS with a layout compatible with boot environments. Mirror and RAID-Z supported.
= Setting up  Alpine Linux using ZFS with a pool that uses ZFS' native encryption capabilities =


Except EFI system partition and boot pool <code>/boot</code>, everything is encrypted. Root pool is encrypted with ZFS native encryption and swap partition is encrypted with dm-crypt.
== Download ==


To do an unencrypted setup, simply omit <code>-O keylocation -O keyformat</code> when creating root pool.
Download the '''extended''' release from https://www.alpinelinux.org/downloads/ as only it contains the zfs kernel mods at the time of this writing (2020.07.10)


= Useful links =
Write it to a USB and boot from it.


*[https://openzfs.github.io/openzfs-docs/Getting%20Started/ OpenZFS Getting Started]
== Initial setup ==


Run the following


= Notes =
    setup-alpine


UEFI is required. Supports single disk & multi-disk (stripe, mirror, RAID-Z) installation.
Answer all the questions, and hit ctrl-c when promted for what disk you'd like to use.


'''Existing data on target disk(s) will be destroyed.'''
== OPTIONAL ==


= Preparation =
This section is optional and it assumes internet connectivity. You may enable sshd so you can ssh into the box and copy and paste the rest of the commands into my terminal window from these instructions.


== Setup live environment ==
Edit `/etc/ssh/sshd_config` and search for `Permit`. Change the value after `PermitRootLogin` to read `yes`


Download the '''''extended''''' release from https://www.alpinelinux.org/downloads/, as only this version is shipped with ZFS kernel module. Alpine Linux can not load kernel module in live.
save and exit to shell. Run `service sshd restart`


Run the following command to setup the live environment, use default <code>none</code> option when asked about disks.
Now you can ssh in as root. Do not forget to go back and comment this line out when you're done since it will be enabled on your resulting machine. You will be reminded again at the end of this doc.


<pre>setup-alpine</pre>
== Add needed packages  ==
Settings given here will be copied to the target system later by <code>setup-disk</code>.


== Install system utilities ==
    apk add zfs sfdisk e2fsprogs syslinux


Install and setup<code>eudev</code> (a port of systemd <code>udev</code> by gentoo) to get block device names.
== Create our partitions ==


<pre>apk update
We're assuming `/dev/sda` here and in the rest of the document but you can use whatever you need to. To see a list, type: `sfdisk -l`
apk add eudev sgdisk grub-efi zfs
modprobe zfs
setup-udev</pre>
= Variables =


In this step, we will set some variables to make our installation process easier.
    echo -e "/dev/sda1: start=1M,size=100M,bootable\n/dev/sda2: start=101M" | sfdisk --quiet --label dos /dev/sda


<pre>DISK=/dev/disk/by-id/ata-HXY_120G_YS</pre>
== Create device nodes ==
Use unique disk path instead of <code>/dev/sda</code> to ensure the correct partition can be found by ZFS.


Other variables
    mdev -s


<pre>TARGET_USERNAME='your username'
== Create the /boot filesystem ==
ENCRYPTION_PWD='your root pool encryption password, 8 characters min'
TARGET_USERPWD='user account password'</pre>
Create a mountpoint


<pre>MOUNTPOINT=`mktemp -d`</pre>
    mkfs.ext4 /dev/sda1
Create a unique suffix for the ZFS pools: this will prevent name conflict when importing pools on another Root on ZFS system.


<pre>poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)</pre>
== Create the root filesystem using zfs ==
= Partitioning =


For a single disk, UEFI installation, we need to create at lease 3 partitions: - EFI system partition - Boot pool partition - Root pool partition Since GRUB only partially support ZFS, many features needs to be disabled on the boot pool. By creating a separate root pool, we can then utilize the full potential of ZFS.
    modprobe zfs
    zpool create -f -o ashift=12 \
        -O acltype=posixacl -O canmount=off -O compression=lz4 \
        -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \
        -O encryption=aes-256-gcm -O keylocation=prompt -O keyformat=passphrase \
        -O mountpoint=/ -R /mnt \
        rpool /dev/sda2


Clear the partition table on the target disk and create EFI, boot and root pool parititions:
You will have to enter your passphrase at this point. Choose wisely, as your passphrase is most likely [https://gitlab.com/cryptsetup/cryptsetup/wikis/FrequentlyAskedQuestions#5-security-aspects the weakest link in this setup].


<pre>sgdisk --zap-all $DISK
A few notes on the options supplied to zpool:
sgdisk -n1:0:+512M -t1:EF00 $DISK
sgdisk -n2:0:+2G $DISK        # boot pool
sgdisk -n3:0:0 $DISK          # root pool</pre>
If you want to use a multi-disk setup, such as mirror or RAID-Z, partition every target disk with the same commands above.


== Optional: Swap partition ==
- `ashift=12` is recommended here because many drives today have 4KiB (or larger) physical sectors, even though they present 512B logical sectors


<code>Swap</code> support on ZFS is also problematic, therefore it is recommended to create a separate Swap partition if needed. This guide will cover the creation of a separate swap partition.(can not be used for hibernation since the encryption key is discarded when power off.)
- `acltype=posixacl` enables POSIX ACLs globally


If you want to use swap, reserve some space at the end of disk when creating root pool:
- `normalization=formD` eliminates some corner cases relating to UTF-8 filename normalization. It also enables `utf8only=on`, meaning that only files with valid UTF-8 filenames will be accepted.


<pre>sgdisk -n3:0:-8G $DISK        # root pool, reserve 8GB for swap at the end of the disk
- `xattr=sa` vastly improves the performance of extended attributes, but is Linux-only. If you care about using this pool on other OpenZFS implementation don't specify this option.
sgdisk -n4:0:0 $DISK          # swap partition</pre>
= Create boot and root pool =


As mentioned above, ZFS features need to be selectively enabled for GRUB. All available features are enabled when no <code>feature@</code> is supplied.
After completing this, confirm that the pool has been created:


Here we explicitly enable those GRUB can support.
    # zpool status


<pre>zpool create \
Should return something like:
  -o ashift=12 -d \
  -o feature@async_destroy=enabled \
  -o feature@bookmarks=enabled \
  -o feature@embedded_data=enabled \
  -o feature@empty_bpobj=enabled \
  -o feature@enabled_txg=enabled \
  -o feature@extensible_dataset=enabled \
  -o feature@filesystem_limits=enabled \
  -o feature@hole_birth=enabled \
  -o feature@large_blocks=enabled \
  -o feature@lz4_compress=enabled \
  -o feature@spacemap_histogram=enabled \
  -O acltype=posixacl -O canmount=off -O compression=lz4 \
  -O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
  -O mountpoint=/boot -R $MOUNTPOINT \
  bpool_$poolUUID $DISK-part2</pre>
Nothing is stored directly under bpool and rpool, hence <code>canmount=off</code>. The respective <code>mountpoint</code> properties are more symbolic than practical.


For root pool all available features are enabled by default
      pool: rpool
    state: ONLINE
      scan: none requested
    config:


<pre>echo $ENCRYPTION_PWD | zpool create \
        NAME        STATE    READ WRITE CKSUM
  -o ashift=12 \
        rpool      ONLINE      0    0    0
  -O encryption=aes-256-gcm \
          sda2      ONLINE      0    0    0
  -O keylocation=prompt -O keyformat=passphrase \
  -O acltype=posixacl -O canmount=off -O compression=lz4 \
  -O dnodesize=auto -O normalization=formD -O relatime=on \
  -O xattr=sa -O mountpoint=/ -R $MOUNTPOINT \
  rpool_$poolUUID $DISK-part3</pre>
== For multi-disk ==


For mirror:
    errors: No known data errors


<pre>zpool create \
== Create the required datasets and mount root ==
  ... \
  bpool_$poolUUID mirror \
  /dev/disk/by-id/target_disk1-part2 \
  /dev/disk/by-id/target_disk2-part2
zpool create \
  ... \
  rpool_$poolUUID mirror \
  /dev/disk/by-id/target_disk1-part3 \
  /dev/disk/by-id/target_disk2-part3</pre>
For RAID-Z, replace mirror with raidz, raidz2 or raidz3.


= Create system datasets =
    zfs create -o mountpoint=none -o canmount=off rpool/ROOT
    zfs create -o mountpoint=legacy rpool/ROOT/alpine
    mount -t zfs rpool/ROOT/alpine /mnt/


This layout is intended to separate root file system from persistent files.
== Mount the `/boot` filesystem ==


<pre>zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
    mkdir /mnt/boot/
zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
    mount -t ext4 /dev/sda1 /mnt/boot/
zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
zfs create -o mountpoint=legacy -o canmount=noauto bpool_$poolUUID/BOOT/default
zfs mount rpool_$poolUUID/ROOT/default
mkdir $MOUNTPOINT/boot
mount -t zfs bpool_$poolUUID/BOOT/default $MOUNTPOINT/boot
# ash, default with busybox, does not support array
# this is word splitting
d='usr var var/lib'
for i in $d; do zfs create  -o canmount=off rpool_$poolUUID/ROOT/default/$i; done
d='srv usr/local'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/$i; done
d='log spool tmp'
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/$i; done
zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
zfs create rpool_$poolUUID/HOME/default/$TARGET_USERNAME</pre>
Depending on your application, separate datasets need to be created for folders inside <code>/var/lib</code>(not itself!)


Here we create several folders for persistent (shared) data, like we just did for <code>/home</code>.
=== Enable ZFS' services ===


<pre>d='libvirt lxc docker'
    rc-update add zfs-import sysinit
for i in $d; do zfs create rpool_$poolUUID/ROOT/default/var/lib/$i; done</pre>
    rc-update add zfs-mount sysinit
<code>lxc</code> is for Linux container, <code>libvirt</code> is for storing virtual machine images, etc.


= Format and mount EFI partition =
== Install Alpine Linux ==


Here we use <code>/boot/efi</code> as the mountpoint, which is default for GRUB.
    setup-disk /mnt
    dd if=/usr/share/syslinux/mbr.bin of=/dev/sda # write mbr so we can boot


<pre>mkfs.vfat -n EFI $DISK-part1
mkdir $MOUNTPOINT/boot/efi
mount -t vfat $DISK-part1 $MOUNTPOINT/boot/efi # need to specify file system</pre>
= System installation =


== Preparation ==
== Reboot and enjoy! ==


GRUB will not find the correct path of root device without ZPOOL_VDEV_NAME_PATH=1.
😉


<pre>export ZPOOL_VDEV_NAME_PATH=1</pre>
'''NOTE:'''
<code>setup-disk</code> refuse to run on ZFS by default, we need to add ZFS to the supported filesystem array.
If you went with the optional step, be sure to disable root login after you reboot.
 
<pre>sed -i 's|supported="ext|supported="zfs ext|g' /sbin/setup-disk</pre>
== setup-disk ==
 
Run <code>setup-disk</code> to install system to target disk.
 
<pre>BOOTLOADER=grub USE_EFI=y setup-disk -v $MOUNTPOINT</pre>
Note that grub-probe will still fail despite <code>ZPOOL_VDEV_NAME_PATH=YES</code> variable set above. We will deal with this later inside chroot.
 
== Chroot ==
 
<pre>m='dev proc sys'
for i in $m; do mount --rbind /$i $MOUNTPOINT/$i; done
chroot $MOUNTPOINT /usr/bin/env TARGET_USERPWD=$TARGET_USERPWD TARGET_USERNAME=$TARGET_USERNAME poolUUID=$poolUUID /bin/sh</pre>
=== Finish GRUB installation ===
 
As GRUB installation failed half-way in [[#Run setup-disk]], we will finish it here.
 
Apply fix:
 
<pre>echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile</pre>
Reload
 
<pre>source /etc/profile</pre>
==== GRUB fails to detect the ZFS filesystem of /boot with BusyBox stat ====
 
<pre>apk add coreutils</pre>
==== Missing root pool ====
 
GRUB will fail to detect rpool if rpool has unsupported features, use the following workaround:
 
<pre>sed -i "s|rpool=.*|rpool=\`zdb -l \${GRUB_DEVICE} \| grep -E '[[:blank:]]name' \| cut -d\\\' -f 2\`|"  /etc/grub.d/10_linux</pre>
This replaces GRUB rpool name detection.
 
==== Generate grub.cfg ====
 
After applying fixes, finally run
 
<pre>grub-mkconfig -o /boot/grub/grub.cfg</pre>
=== Importing pools on boot ===
 
<code>zpool.cache</code> will be added to initramfs and zpool command will import pools contained in this cache.
 
System will fail to boot without this.
 
<pre>zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID</pre>
=== Initramfs ===
 
<code>mkinitfs</code> included in stable Alpine Linux has bugs, see [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/77 1] and [https://gitlab.alpinelinux.org/alpine/mkinitfs/-/merge_requests/76 2].
 
==== Add eudev hook and rebuild ====
 
Add <code>eudev</code> to <code>/etc/mkinitfs/mkinitfs.conf</code>.
 
<pre>echo 'features="ata base eudev ide scsi usb virtio nvme zfs"' > /etc/mkinitfs/mkinitfs.conf
# order of features is important! this order is tested</pre>
Rebuild initramfs with
 
<pre>mkinitfs $(ls -1 /lib/modules/)</pre>
 
=== Mount datasets at boot ===
 
<pre>rc-update add zfs-mount sysinit</pre>
 
=== Add user ===
 
<pre>adduser -s /bin/sh -H -D -h /home/$TARGET_USERNAME $TARGET_USERNAME
chown -R $TARGET_USERNAME /home/$TARGET_USERNAME
echo "$TARGET_USERNAME:$TARGET_USERPWD" | chpasswd</pre>
Root account is accessed via <code>su</code> command with root password.
 
=== Boot environment manager ===
 
[https://gitlab.com/m_zhou/bieaz bieaz] is a simple boot environment management shell script with GRUB integration.
 
It has been submitted to aports, see [https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/16406 this merge request]. Should be available in edge/test soon.
 
=== Optional: Enable encrypted swap partition ===
 
Install <code>cryptsetup</code>
 
<pre>apk add cryptsetup</pre>
Edit the <code>/etc/mkinitfs/mkinitfs.conf</code> file and append the <code>cryptsetup</code> module to the front of zfs. Add relevant lines in <code>fstab</code> and <code>crypttab</code>. Replace <code>$DISK</code> with actual disk.
 
<pre>echo swap  $DISK-part4 /dev/urandom    swap,cipher=aes-cbc-essiv:sha256,size=256 >> /etc/crypttab
echo /dev/mapper/swap  none    swap    defaults    0  0 >> /etc/fstab</pre>
Rebuild initramfs with <code>mkinitfs</code>.
 
= Finish installation =
 
Take a snapshot for the clean installation for future use and export all pools.
 
<pre>exit
zfs snapshot -r rpool_$poolUUID/ROOT/default@install
zfs snapshot -r bpool_$poolUUID/BOOT/default@install</pre>
Pools must be exported before reboot, or they will fail to be imported on boot.
 
<pre>mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID</pre>
= Reboot =
 
<pre>reboot</pre>
= Recovery in Live environment =
 
Boot Live environment (extended release) and repeat [[#preparation|Preparation]]
 
Create a mount point and store encryption password in a variable:
 
<pre>MOUNTPOINT=`mktemp -d`
ENCRYPTION_PWD='YOUR DISK ENCRYPTION PASSWORD, 8 MINIMUM'</pre>
Find the unique UUID of your pool with
 
<pre>zpool import</pre>
Import rpool without mounting datasets: <code>-N</code> for not mounting all datasets; <code>-R</code> for alternate root.
 
<pre>poolUUID=abc123
zpool import -N -R $MOUNTPOINT rpool_$poolUUID</pre>
Load encryption key
 
<pre>echo $ENCRYPTION_PWD | zfs load-key -a</pre>
As <code>canmount=noauto</code> is set for <code>/</code> dataset, we have to mount it manually. To find the dataset, use
 
<pre>zfs list rpool_$poolUUID/ROOT</pre>
Mount <code>/</code> dataset
 
<pre>zfs mount rpool_$UUID/ROOT/$dataset</pre>
Mount other datasets
 
<pre>zfs mount -a</pre>
Import bpool
 
<pre>zpool import -N -R $MOUNTPOINT bpool_$UUID</pre>
Find and mount the <code>/boot</code> dataset, same as above.
 
<pre>zfs list bpool_$UUID/BOOT
mount -t zfs bpool_$UUID/BOOT/$dataset $MOUNTPOINT/boot # legacy mountpoint</pre>
Chroot
 
<pre>mount --rbind /dev  $MOUNTPOINT/dev
mount --rbind /proc $MOUNTPOINT/proc
mount --rbind /sys  $MOUNTPOINT/sys
chroot $MOUNTPOINT /bin/sh</pre>
After chroot, mount <code>/efi</code>
 
<pre>mount /boot/efi</pre>
After fixing the system, don't forget to umount and export the pools:
 
<pre>mount | grep -v zfs | tac | grep $MOUNTPOINT | awk '{print $3}' | \
xargs -i{} umount -lf {}
zpool export bpool_$poolUUID
zpool export rpool_$poolUUID</pre>

Revision as of 14:19, 7 January 2021

Setting up Alpine Linux using ZFS with a pool that uses ZFS' native encryption capabilities

Download

Download the extended release from https://www.alpinelinux.org/downloads/ as only it contains the zfs kernel mods at the time of this writing (2020.07.10)

Write it to a USB and boot from it.

Initial setup

Run the following 

   setup-alpine

Answer all the questions, and hit ctrl-c when promted for what disk you'd like to use.

OPTIONAL

This section is optional and it assumes internet connectivity. You may enable sshd so you can ssh into the box and copy and paste the rest of the commands into my terminal window from these instructions.

Edit `/etc/ssh/sshd_config` and search for `Permit`. Change the value after `PermitRootLogin` to read `yes`

save and exit to shell. Run `service sshd restart`

Now you can ssh in as root. Do not forget to go back and comment this line out when you're done since it will be enabled on your resulting machine. You will be reminded again at the end of this doc.

Add needed packages

   apk add zfs sfdisk e2fsprogs syslinux

Create our partitions

We're assuming `/dev/sda` here and in the rest of the document but you can use whatever you need to. To see a list, type: `sfdisk -l` 

   echo -e "/dev/sda1: start=1M,size=100M,bootable\n/dev/sda2: start=101M" | sfdisk --quiet --label dos /dev/sda

Create device nodes

   mdev -s

Create the /boot filesystem

   mkfs.ext4 /dev/sda1

Create the root filesystem using zfs

   modprobe zfs
   zpool create -f -o ashift=12 \
       -O acltype=posixacl -O canmount=off -O compression=lz4 \
       -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \
       -O encryption=aes-256-gcm -O keylocation=prompt -O keyformat=passphrase \
       -O mountpoint=/ -R /mnt \
       rpool /dev/sda2

You will have to enter your passphrase at this point. Choose wisely, as your passphrase is most likely the weakest link in this setup.

A few notes on the options supplied to zpool:

- `ashift=12` is recommended here because many drives today have 4KiB (or larger) physical sectors, even though they present 512B logical sectors

- `acltype=posixacl` enables POSIX ACLs globally

- `normalization=formD` eliminates some corner cases relating to UTF-8 filename normalization. It also enables `utf8only=on`, meaning that only files with valid UTF-8 filenames will be accepted.

- `xattr=sa` vastly improves the performance of extended attributes, but is Linux-only. If you care about using this pool on other OpenZFS implementation don't specify this option.

After completing this, confirm that the pool has been created: 

   # zpool status

Should return something like: 

     pool: rpool
    state: ONLINE
     scan: none requested
   config:

       NAME        STATE     READ WRITE CKSUM
       rpool       ONLINE       0     0     0
         sda2      ONLINE       0     0     0

   errors: No known data errors

Create the required datasets and mount root

   zfs create -o mountpoint=none -o canmount=off rpool/ROOT
   zfs create -o mountpoint=legacy rpool/ROOT/alpine
   mount -t zfs rpool/ROOT/alpine /mnt/

Mount the `/boot` filesystem

   mkdir /mnt/boot/
   mount -t ext4 /dev/sda1 /mnt/boot/

Enable ZFS' services

   rc-update add zfs-import sysinit
   rc-update add zfs-mount sysinit

Install Alpine Linux

   setup-disk /mnt
   dd if=/usr/share/syslinux/mbr.bin of=/dev/sda # write mbr so we can boot


Reboot and enjoy!

😉

NOTE: If you went with the optional step, be sure to disable root login after you reboot.

Retrieved from "https://wiki.alpinelinux.org/w/index.php?title=Root_on_ZFS_with_native_encryption&oldid=18521"