If you need urgent consulting help click here

Zephyr Audio DSP Development on Chromebooks

The Audio DSP on Intel Chromebooks is configured to use the SOF “Community” key for firmware signing, and can therefore accept arbitrary user-developed firmware like Zephyr applications (of which SOF is one), including the Zephyr samples and test suite.

Initial TGL Chromebook Setup

(These instructions were written specifically to the Asus Flip CX5 device code named “delbin”. But they should be reasonably applicable to any recent Intel device.)

Power the device on and connect it to a wireless network. It will likely want to download a firmware update (mine did). Let this finish first, to ensure you have two working OS images.

Enable Developer Mode

Power the device off (menu in lower right, or hold the power button on the side)

Hold Esc + Refresh (the arrow-in-a-circle “reload” key above “3”) and hit the power key to enter recovery mode. Note: the touchscreen and pad don’t work in recovery mode, use the arrow keys to navigate.

Select “Advanced Options”, then “Enable Developer Mode” and confirm that you really mean it. Select “Boot from Internal Storage” at the bootloader screen. You will see this screen every time the machine boots now, telling you that the boot is unverified.

Wait while the device does the required data wipe. My device takes about 15 minutes to completely write the stateful partition. On reboot, select “Boot from Internal Storage” again and set it up (again) with Google account.

Make a Recovery Drive

You will at some point wreck your device and need a recovery stick. Install the Chromebook Recovery Utility from the Google Web Store and make one.

You can actually do this on any machine (and any OS) with Chrome installed, but it’s easiest on the Chromebook because it knows its device ID (for example “DELBIN-XHVI D4B-H4D-G4G-Q9A-A9P” for the Asus Tiger Lake board). Note that recovery, when it happens, will not affect developer mode or firmware settings but it will wipe out the root filesystem and /usr/local customizations you have made. So plan on a strategy that can tolerate data loss on the device you’re messing with!

Make the root filesystem writable

For security, ChromeOS signs and cryptographically verifies (using Linux’s dm-verity feature) all access to the read-only root filesystem. Mucking with the rootfs (for example, to install modules for a custom kernel) requires that the dm-verity layer be turned off:

First open a terminal with Ctrl-Alt-T. Then at the “crosh> ” prompt issue the “shell” command to get a shell running as the “chronos” user. Finally (in developer mode) a simple “sudo su -” will get you a root prompt.

crosh> shell
chronos@localhost / $ sudo su -
localhost ~ #

Now you need to turn of signature verification in the bootloader (because obviously we’ll be breaking whatever signature existed). Note that signature verification is something done by the ROM bootloader, not the OS, and this setting is a (developer-mode-only) directive to that code:

cros# crossystem dev_boot_signed_only=0

(Note: for clarity, commands in this document entered at the ChromeOS core shell will be prefixed with a hostname of cros.)

Next you disable the validation step:

cros# /usr/share/vboot/bin/make_dev_ssd.sh --remove_rootfs_verification

THIS COMMAND WILL FAIL, give you an error that you are changing the setting for the entire running system, and suggest an alternative “–partitions X” argument to use that modifies only the currently used partition. Run that modified command, then reboot.

After rebooting, you will notice that your chromebook boots with the raw storage device (e.g. /dev/nvme0n1p5) mounted as root and not the “dm-0” verity device, and that the rootfs is read-write.

Note: What this command actually does is modify the command line of the installed kernel image (it saves a backup in /mnt/stateful_partition/cros_sign_backups) so that it specifies “root=<guid>” and not “root=dm-0”. It does seem to leave the other verity configuration in place though, it just doesn’t try to mount the resulting (now-invalid!) partition.

Metanote: The astute will note that we’re probably going to throw this kernel out, and that we could probably have just edited the command line of the new kernel instead of flashing and rebooting into this modified one. But that’s too many balls to juggle at once for me.

Enable ChromeOS SSH

Once you are booted with a writable partition, you can turn on the built-in ssh server with:

cros# /usr/libexec/debugd/helpers/dev_features_ssh

By default neither the “chronos” user nor root accounts have passwords, so unless you want to type a ssh key in by hand, you probably want to set a password for the first login (before you run ssh-copy-id, of course):

cros# passwd

Now ssh into the chromebook and add your key to .ssh/authorized_keys as you do for any Linux system.

Install Crouton

The Zephyr integration tools require a proper Linux environment and won’t run on ChromeOS’s minimal distro. So we need to install a Linux personality. DO NOT bother installing the “Linux Development Environment” (Crostini) from the ChromeOS Developer settings. This personality runs inside a VM, where our tools need access to the real kernel running on the real hardware. Instead install Crouton (https://github.com/dnschneid/crouton), which is a community chroot-based personality that preserves access to the real hardware sysfs and /dev filesystem. These instructions install the “cli-extra” package list, there are X11-enabled ones available too if you prefer to work on the device screen directly. See the project page, etc…

At a root shell, grab the installer and run it (note: /usr/local is the only writable filesystem without noexec, you must place the binary there for it to run!):

cros# mkdir -p /usr/local/bin
cros# curl -L https://github.com/dnschneid/crouton/raw/master/installer/crouton \
              > /usr/local/bin/crouton
cros# chmod 755 /usr/local/bin/crouton
cros# crouton -r focal -t cli-extra

Start the Crouton chroot environment:

cros# startcli

Now you are typing commands into the Ubuntu environment. Enable inbound ssh on Crouton, but on a port other than 22 (which is used for the native ChromeOS ssh server). I’m using 222 here (which is easy to remember, and not a registered port in /etc/services):

crouton# apt install iptables openssh-server
crouton# echo "Port 222" >> /etc/ssh/sshd_config
crouton# mkdir /run/sshd
crouton# iptables -I INPUT -p tcp --dport 222 -j ACCEPT
crouton# /usr/sbin/sshd

(As above: note that we have introduced a hostname of “crouton” to refer to the separate Linux personality.)

NOTE: the mkdir, iptables and sshd commands need to be run every time the chroot is restarted. You can put them in /etc/rc.local for convenience. Crouton doesn’t run systemd (because it can’t – it doesn’t own the system!) so Ubuntu services like openssh-server don’t know how to start themselves.

Building and Installing a Custom Kernel

On your build host, grab a copy of the ChromeOS kernel tree. The shipping device is using a 5.4 kernel, but the 5.10 tree works for me and seems to have been backporting upstream drivers such that its main hardware is all quite recent (5-6 weeks behind mainline or so). We place it in the home directory here for simplicity:

dev$ cd $HOME
dev$ git clone https://chromium.googlesource.com/chromiumos/third_party/kernel
dev$ cd kernel
dev$ git checkout chromeos-5.10

(Once again, we are typing into a different shell. We introduce the hostname “dev” here to represent the development machine on which you are building kernels and Zephyr apps. It is possible to do this on the chromebook directly, but not advisable. Remember the discussion above about requiring a drive wipe on system recovery!)

Note: you probably have an existing Linux tree somewhere already. If you do it’s much faster to add this as a remote there and just fetch the deltas – ChromeOS tracks upstream closely.

Now you need a .config file. The Chromebook kernel ships with the “configs” module built which exposes this in the running kernel. You just have to load the module and read the file.

dev$ cd /path/to/kernel
dev$ ssh root@cros modprobe configs
dev$ ssh root@cros zcat /proc/config.gz > .config

You will need to set some custom configuration variables differently from ChromeOS defaults (you can edit .config directly, or use menuconfig, etc…):

  • CONFIG_HUGETLBFS=y - The Zephyr loader tool requires this

  • CONFIG_EXTRA_FIRMWARE_DIR=n - This refers to a build directory

    in Google’s build environment that we will not have.

  • CONFIG_SECURITY_LOADPIN=n - Pins modules such that they will

    only load from one filesystem. Annoying restriction for custom kernels.

  • CONFIG_MODVERSIONS=n - Allow modules to be built and installed

    from modified “dirty” build trees.

Now build your kernel just as you would any other:

dev$ make olddefconfig     # Or otherwise update .config
dev$ make bzImage modules  # Probably want -j<whatever> for parallel build

The modules you can copy directly to the (now writable) rootfs on the device. Note that this filesystem has very limited space (it’s intended to be read only), so the INSTALL_MOD_STRIP=1 is absolutely required, and you may find you need to regularly prune modules from older kernels to make space:

dev$ make INSTALL_MOD_PATH=mods INSTALL_MOD_STRIP=1 modules_install
dev$ (cd mods/lib/modules; tar cf - .) | ssh root@cros '(cd /lib/modules; tar xfv -)'

Pack and Install ChromeOS Kernel Image

The kernel bzImage file itself needs to be signed and packaged into a ChromeOS vboot package and written directly to the kernel partition. Thankfully the tools to do this are shipped in Debian/Ubuntu repositories already:

$ sudo apt install vboot-utils vboot-kernel-utils

Find the current kernel partition on the device. You can get this by comparing the “kernel_guid” command line parameter (passed by the bootloader) with the partition table of the boot drive, for example:

dev$ KPART=`ssh root@cros 'fdisk -l -o UUID,Device /dev/nvme0n1 | \
                           grep -i $(sed "s/.*kern_guid=//" /proc/cmdline \
                                     | sed "s/ .*//") \
                           | sed "s/.* //"'`
dev$ echo $KPART
/dev/nvme0n1p4

Extract the command line from that image into a local file:

dev$ ssh root@cros vbutil_kernel --verify /dev/$KPART | tail -1 > cmdline.txt

Now you can pack a new kernel image using the vboot tooling. Most of these arguments are boilerplate and always the same. The keys are there because the boot requires a valid signature, even though as configured it won’t use it. Note the cannot-actually-be-empty dummy file passed as a “bootloader”, which is a holdover from previous ROM variants which needed an EFI stub.

dev$ echo dummy > dummy.efi
dev$ vbutil_kernel --pack kernel.img --config cmdline.txt \
       --vmlinuz arch/x86_64/boot/bzImage \
       --keyblock /usr/share/vboot/devkeys/kernel.keyblock \
       --signprivate /usr/share/vboot/devkeys/kernel_data_key.vbprivk \
       --version 1 --bootloader dummy.efi --arch x86_64

You can verify this image if you like with “vbutil_kernel –verify”.

Now just copy up the file and write it to the partition on the device:

$ scp kernel.img root@cros:/tmp
$ ssh root@cros dd if=/tmp/kernel.img of=/dev/nvme0n1p4

Now reboot, and if all goes well you will find yourself running in your new kernel.

Wifi Firmware Fixup

On the Tiger Lake Chromebook, the /lib/firmware tree is a bit stale relative to the current 5.10 kernel. The iwlwifi driver requests a firmware file that doesn’t exist, leading to a device with no network. It’s a simple problem, but a catastrophic drawback if uncorrected. It seems to be sufficient just to link the older version to the new name. (It would probably be better to copy the proper version from /lib/firmware from a recent kernel.org checkout.):

cros# cd /lib/firmware
cros# ln -s iwlwifi-QuZ-a0-hr-b0-62.ucode iwlwifi-QuZ-a0-hr-b0-64.ucode

Build and Run a Zephyr Application

Finally, with your new kernel booted, you are ready to run Zephyr code.

Build rimage Signing Tool

First download and build a copy of the Sound Open Firmware “rimage” tool (these instructions put it in your home directory for clarity, but anywhere is acceptable):

dev$ cd $HOME
dev$ git clone https://github.com/thesofproject/rimage
dev$ cd rimage/
dev$ git submodule init
dev$ git submodule update
dev$ cmake .
dev$ make

Copy Integration Scripting to Chromebook

There are two python scripts needed on the device, to be run inside the Crouton environment installed above. Copy them:

dev$ scp boards/xtensa/intel_adsp_cavs15/tools/cavs-fw-v25.py root@crouton:
dev$ scp boards/xtensa/intel_adsp_cavs15/tools/adsplog.py root@crouton:

Build and Sign Zephyr App

Zephyr applications build conventionally for this platform, and are signed with “west flash” with just a few extra arguments. Note that the key in use for the Tiger Lake DSP is the “3k” key from SOF, not the original that is used with older hardware. The output artifact is a “zephyr.ri” file to be copied to the device.

dev$ west build -b intel_adsp_cavs25 samples/hello_world
dev$ west sign --tool-data=~/rimage/config -t ~/rimage/rimage -- \
            -k $ZEPHYR_BASE/../modules/audio/sof/keys/otc_private_key_3k.pem
dev$ scp build/zephyr/zephyr.ri root@crouton:

Run it!

The loader script takes the signed rimage file as its argument. Once it reports success, the application begins running immediately and its console output (in the SOF shared memory trace buffer) can be read by the logging script.

crouton# ./cavs-fw-v25.py zephyr.ri
crouton# ./adsplog.py
Hello World! intel_adsp_cavs25

Misc References

Upstream documentation from which these instructions were drawn:

This page has the best reference for the boot process:

http://www.chromium.org/chromium-os/chromiumos-design-docs/disk-format

This is great too, with an eye toward booting things other than ChromeOS:

https://www.chromium.org/chromium-os/developer-information-for-chrome-os-devices/custom-firmware