How to make a chroot to run a different architecture (e.g. x86 on arm)

If you have an arm computer, you might find that some software is only available for x86_64. One possible solution is to have a full x86_64 virtual machine, but that has quite a lot of overhead from having to emulate the entire system. Fortunately, there is QEMU User space emulation which still uses the host kernel, so there is less overhead. This guide will show you how to make a chroot and enable transparent x86_64 binary execution using qemu and binfmt_misc.

Prerequisites

You should be familiar with the linux command line. This guide is written assuming you have alpine as the main install as well as putting alpine in the chroot.

Also, the commands shown assume you have root privileges. Either use sudo as needed or su first.

Installing necessary software

First, Enable the Community Repository. Next, install the qemu user mode binaries for the architectures you want to use, in this case x86_64 and i386:

apk add qemu-x86_64 qemu-i386

Please note there are the qemu-system-* packages, you don't want those since they are used for full virtual machine emulation.

Then, curl my modified version of the qemu-binfmt script (originally from the qemu-openrc package):

curl -o /etc/init.d/qemu-binfmt 'https://gist.githubusercontent.com/ktprograms/c8ea850abde717376ff7a6f83dbabb2f/raw/facb4ae367d3c7219949f33a566234427b0c2f54/qemu-binfmt'

I prefer to use this script instead of installing the qemu-openrc package since that adds a lot of unneeded dependencies. Right now this only supports x86_64 and i386 but I'll be adding all the architectures soon.

Make the qemu-binfmt script executable:

chmod +x /etc/init.d/qemu-binfmt

Enable the `qemu-binfmt` service and start it

The qemu-binfmt script is an OpenRC init script that registers the binfmt_misc handlers when the service starts and unregisters them when the service stops. Add the service to the default runlevel and start it immediately:

rc-update add qemu-binfmt default
service qemu-binfmt start

Make the chroot folder

To keep it simple, we're just going to use the apk repositories specified on the host system. This guide assumes the $CHROOT environment variable holds the directory where you want to make the chroot. Make the chroot directory and the etc/apk subdirectory (which we'll copy the apk repositories file to):

mkdir -p $CHROOT/etc/apk/

Now copy the host's apk repositories into the chroot folder:

cp /etc/apk/repositories $CHROOT/etc/apk/

You'll also need to copy the resolv.conf file into the chroot so it can have internet access:

cp /etc/resolv.conf $CHROOT/etc/

Install `alpine-base` into the chroot folder

The alpine-base meta package is all that's needed to have a fully working alpine system. The apk invocation used here has a few more options than what you might be used to:

apk add -p $CHROOT --initdb -U --arch x86_64 --allow-untrusted alpine-base

Here's what all the options do:

-p $CHROOT: Sets the root of the install to $CHROOT instead of /
--initdb: Initialize a new package database.
-U: Alias for --cache-max-age 1. Basically means don't install from cache.
--arch x86_64: Install x86_64 packages instead of native packages. Replace this with whichever architecture you want to emulate instead (e.g. --arch x86).
--allow-untrusted: Install packages with untrusted signature or no signature. (This is needed since the keys for different architectures are different than what will be on your host. If you have the correct keys, you can copy them into the $CHROOT/etc/apk/keys/ directory and remove this flag).

Mount the virtual file systems into the chroot

mount -t proc proc $CHROOT/proc/
mount -t sysfs sys $CHROOT/sys/
mount -o bind /dev/ $CHROOT/dev/
mount -o bind /dev/pts/ $CHROOT/dev/pts/
mount -o bind /run $CHROOT/run/

The exciting part: Enter the chroot

If you've followed all the steps, this part should be uneventful. The chroot invocation uses the special command of ash -l to run as a login shell and source /etc/profile:

chroot $CHROOT/ ash -l

Troubleshooting

If you get a ERROR: ${package_name}.post-install: script exited with error 127 while trying to add the alpine-base, that normally means that qemu binfmt_misc hasn't been setup properly. Check if the qemu-$arch package has been installed and see if the binfmt_misc service is running. You can also see if there is a qemu-$arch entry in /proc/sys/fs/binfmt_misc/.

Programs that use Netlink (e.g. ip addr) don't seem to work properly since qemu doesn't emulate the Netlink protocol properly.

References

https://ownyourbits.com/2018/06/13/transparently-running-binaries-from-any-architecture-in-linux-with-qemu-and-binfmt_misc/ (see the section titled Emulating full ARM rootfs)