Embedded Linux Completely from Scratch

Each node of my cluster computer has 2GiB of memory. RAM is at a premium. Each node is nameless and stateless like an AWS Lambda (or Rancher cattle, or Saltstack Minion), so the entire OS must reside in scarce memory along with the user space for jobs like map-reduce operations.

Having explored minimal Debian, Alpine Linux, and even RancherOS¹ the most rewarding conclusion is to put in the time to learn to compile Linux from scratch for ARM64 (also called ARMv8 and AArch64).

Here are the talking points:

1. Kernel, Initramfs, DTBs
2. Deep-Learning Settings
3. U-Boot and iPXE Bootloaders
4. User Processes and Daemons
5. Network Bootloading on Real Hardware
6. Gotchas and More Tips
7. Conclusion

Creating an Embedded Linux Distro

Here is a checklist of items needed to build and network-boot a Linux distro into RAM:

• Bootloader stage 1 (U-Boot in SPI flash)
• Bootloader stage 2 (iPXE for HTTP transport)
• Linux kernel and drivers
• Device hardware tree (DTB)
• Initramfs – RAM filesystem
• User-space daemons
• PXE server

The plan is first to compile the Linux kernel and initramfs and run them directly in a virtual environment (QEMU). Next, compile U-Boot and iPXE bootloaders. Then, we’ll see if I can bootload the distro in QEMU. I’ll then add a daemon that makes a light blink to show boot was successful. Penultimately, I’ll set up Dnsmasq and a TFTP server to boot some real hardware. Finally, I’ll try to optimize the kernel image size.

Part 1 – Kernel, Initramfs, DTBs

The first milestone is to compile the kernel, the RAM filesystem, and device trees (DTBs) from Git source.

Cross-Compiled Linux From Scratch (CLFS)

Has anyone done this before? Using the work of CLFS and Cortex A53 hackers, I first attempt to reproduce their written steps in a Dockerfile.

Most of this work is going to take place in a Dockerfile because one’d prefer not to wget large archives each time the compilation fails while learning. Docker can cache previous steps, including previous downloads and build stages. The first six chapters of the CLFS guide can be scripted (with my updates added, like using arm64 instead of 32-bit arm, and using armv8-a instead of… nothing), so here is my initial Dockerfile (remember, please do not use this) following the old CLFS guide to getting to the interactive Linux kernel compilation:

At first blush, this appears to be succeeding because next, we’ll configure the Linux kernel.

Linux Kernel Configuration

When the Linux kernel configuration menu appeared for the first time after a day of writing the above Dockerfile, I had no idea what I was doing. Do I need ARM Accelerated Cryptographic Algorithms? What is “Accelerated scalar and SIMD Poly1305 hash implementations”? What is the Kernel Hacking sub-menu? Should I debug “Oops, Lockups, and Hangs”? Do I need the new Google Firmware Drivers? What is all this?

Fortunately, there is a body of work from some Cortex A53 (Pine64) kernel hackers²³, though abandoned since 2018. What they left for me to reverse engineer is two kernel configurations for the same hardware I’m targeting with nominal settings and minimal settings. One line item I noticed is CONFIG_KERNEL_MODE_NEON=y. What is this? According to Arm⁴,

Neon technology is a packed SIMD architecture. Neon registers are considered as vectors of elements… Neon can also accelerate signal processing algorithms and functions to speed up applications such as audio and video processing, voice and facial recognition, computer vision, and deep learning.

That sounds wonderful – let’s keep that, right? I admit I have no idea what to include, so I’ll include all I can for now and whittle down features as I learn about them.

Cryptography and Deep-Learning Settings

Let’s take a moment to see what is under the hood in the Cortex A53 CPU since CPU instruction sets came up.

The CLFS guide suggests using CLFS_ARM_ARCH=armv7. I know the 64-bit Arm chips are at least armv8. Digging around the net I came to an Arm article that piqued my curiosity – it made reference to minor versions of the Armv8 architecture: armv8.1-a, armv8.2-a, etc. that add vector support and SIMD instructions for deep learning. My Cortex A53 chips are ubiquitous, but they only use the armv8.0-a instruction set. Adding to that, we can select extensions to the instruction set:

• +crc – The Cyclic Redundancy Check (CRC) instructions.
• +simd – The ARMv8-A Advanced SIMD and floating-point instructions.
• +crypto – The cryptographic instructions.
• +sb – Speculation Barrier Instruction⁵.
• +predres – Execution and Data Prediction Restriction Instructions.
• +nofp – Disable the floating-point, Advanced SIMD, and cryptographic instructions.

How do we want to tune the compiled code? Looking at the above, if my tuned code is aimed at processing whole-number time-series data (and it is), then I want floating-points off to speed up whole-number arithmetic. However, if performing ML (and I will), then floats are critical to have in hardware. Decisions, decisions. For now, I will leave hardware floats on and tweak these in the future. Regardless, this is valuable information about tuned instruction sets for specific work.

AArch64 Musl Cross-Compile Toolchain

Trying to keep this a manageable project, I forked and updated an existing toolchain project: https://github.com/ericdraken/musl-cross-make. This is the latest, stable cross-compile build tool using Musl for ARM64 with SHA1 download verifications. With this, I’ll continue on to build the kernel again, next.

These tools compiled beautifully, so let’s update the initial Dockerfile to use the new toolchain. That should work, right?

Enter Buildroot – A Cross-Compilation Embedded-Linux Build Tool

After manually going through all the CLFS steps to compile a Linux distro, it turns out there is a project by the team at the Buildroot Association to do exactly my goal: compile a kernel and rootfs using the latest Linux source, musl, GCC 10+, and all on AArch64 for Cortex A53 and with the device trees for my SoC boards. Just beautiful. Here, I’ll offer my automated and optimized scripts for building complete embedded Linux “distros” for my cluster computer.

Buildroot has a handy Xconfig GUI to visually edit the configuration settings, but most of the time I will edit them in the shell by hand and for faster diffing between tweaks.

Increase Build Speed

What works great is to use a Docker image with host build tools, and when I run it for a build, I mount a local folder to /downloads, and a wipeable Docker volume to /ccache. This way compilation is greatly sped up with caching, and downloads are already present. This saves downloading hundreds of megabytes per build.

Compiling the toolchain takes about 70% of the build time. Let’s cache that too. Buildroot suggests to compile a toolchain SDK, save to a tarball, and use it again as an external toolchain.

Additionally, I like to blow away my entire work folder on each run. Why? Believe me, runs from previous builds are additive, not selective. I like to maintain individual config files and a pristine git folder of Buildroot. After cleaning, I copy those back to a work folder. Clean.

External Toolchain

This is the most important section. I’ve experimented with several toolchains like Linaro, ARM’s, glibc, musl, and others. I prefer musl because builds compile smaller, and it feels like I’m working with Alpine Docker images.

Building my own toolchain, I can use the latest kernel headers, GCC, and musl version. Plus I can tune the builds for performance and size closely. Nice.

Linux Kernel Settings

We can compile the latest kernel from source. However, configuring the kernel is amazingly complex – there are far too many options and tri-state drivers to choose from for all kinds of hardware. What I’ve learned is to include all possible USB and USB HID modules because soon I will control a Blinkstick from this forthcoming tiny distro. One way to go is to select all options and drivers and bloat the kernel image like a blowfish, then whittle away drivers once all hardware works outright.

Here is a sample of those complex build options again:

BusyBox

Space is at a premium, but only a subset of shell utilities is required in embedded systems. BusyBox handles most of what an embedded system needs. Some features are missing, like sort -h (human-readable number sort), but the space savings greatly make up for these inconveniences at around ~720KiB for one binary that does it all.

BusyBox must also be compiled from source and can be heavily configured. Again, include everything so that userspace apps work, then remove features, build, and bisect until an optimal BusyBox binary is reached.

Device Trees (DTBs)

This is an interesting one: the Linux kernel can compile the DTBs, or not, or the U-Boot bootloader can compile them. The DTBs (FIT) can be baked into the Kernel, specified externally with “bootargs”, or referenced from memory when U-Boot loads the DTB independently. I spent a good deal of time experimenting here because with the wrong settings, the kernel doesn’t display anything to the console. Fun.

QEMU Kernel Emulation

This was a beast. I put QEMU into a Docker image to protect my host from bloat, then I figured out how to pipe USB devices into the container, safely, without the dangerous --privileged flag using Cgroups. Templating this script to be included with each kernel build, here is a sample QEMU script to illustrate some useful flags. See near the end of the script below for the QEMU flags.

Success. The kernel and RAM filesystem boot up to an interactive shell.

With custom and maintainable build scripts, the result is stripped config files (defconfigs), the device-tree binaries (DTBs), the initramfs and Image files, and even a handy Graphviz graph of the dependencies.

Here is a sample of a Buildroot dependency graph.

Part 2 – U-Boot and iPXE Bootloaders

The next milestone is to compile mainline U-Boot and iPXE, the latter allowing booting over HTTP.

U-Boot – Bare-Metal Bootloader

Credit where credit is due, I reverse-engineered this project to figure out how to flash SPI memory with U-Boot. With this information, I turned to mainline U-Boot to make some custom build scripts. Now I can keep up with mainline releases. Here are my U-Boot settings for my Pine64 modules with detailed logging enabled:

The embedded commands to flash the SPI with U-Boot are quite small (named flash-spi.cmd):

Then, the most important addition to the mainline U-Boot Makefile is some appended rules to build the SPI flasher:

With this technique of appending Makefile targets, we can continue to use mainline and simply append my SPI-flasher targets on new releases, then build with make all && make spi-flasher.

iPXE – Secondary HTTP Bootloader

iPXE (Pre-eXecution Environment) is overkill, but as a secondary bootloader that can load the kernel and initramfs over HTTP(S) with a lot of flexibility, it’s wonderful. If we need to drop into the iPXE shell, there are several handy networking and debugging tools available as well. Plus, we can chainload over HTTP meaning iPXE can ask for bootscripts to make further HTTP network-boot requests. The responses can be dynamic since the requests call an HTTP server. Imagination, activate.

Not everything is rainbows, as iPXE uses interrupts, but U-Boot (which launches iPXE) runs on a single thread, so hacking is needed. Also, several ARM64 features have not been added to iPXE (they exist in x86-64) and from reading the room in the message boards they will not be added, so, more hacking. Honestly, I was a hair away from abandoning iPXE and bootloading over NFS, but I persisted. Maybe this will be useful to others.

QEMU PXE Bootload Emulation

This is slightly easier, and good news: QEMU has a built-in TFTP server (through SLiRP), so with a few modifications to my early QEMU launcher script, we can demonstrate bootloading in QEMU. One caveat is that the bootloader script must be baked into the iPXE binary because we don’t have a Dnsmasq server, but just for QEMU.

iPXE bootloader script for QEMU:

Modified QEMU launcher script:

The relevant lines are:

Part 3 – User Processes and Daemons

The next milestone is to add background daemons to run on boot. One could be a SaltStack listener. Another could be a performance monitor (RAM, temperature, etc.). But, how to add these in Buildroot?

Let’s have some fun and install a background process that lights a USB-connected LED green when Linux has booted, and goes dark if Linux crashes or hasn’t booted yet, or when shutdown gracefully. Let’s call this the Blinkstick daemon. This is an opportunity to explore compiling custom modules with Buildroot.

Add Modules to Buildroot

Needed: Python3, libusb1.0, a python controller script, and all the kernel USB drivers we can get. The fun part is that all these prerequisites can be added as modules within the Buildroot configuration menu. Let’s see how to do that. For example,

Then, in the main Buildroot Config.in, we can add these menu items to include the Blinkstick module:

I’m quite proud of this next part. With Buildroot, you can add Make instructions for additional modules. In the case of Blinkstick, it was abandoned at Python2, and clearly written in Windows based on the line endings (CRLF). Let’s make it work with Python3, on Linux, and fix a critical bug where libusb cannot be found when using musl (our toolchain).

In this makefile fragment, we convert the line endings to Linux (LF) and hardcode the libusb-1.0.so path due to incompatibilities with Linux and musl. This is just a proof-of-concept; I’m determined to rewrite this controller in C++.

BusyBoxy Init

Finally, with BusyBox’s inittab, we can write a single line of code, or place a script in an init.d folder.

The invoked script is then:

A background daemon that keeps a green light alive as long as the kernel is running is now resident.

Part 4 – Network Bootloading on Real Hardware

It’s not wizardry, but yes you can have two DHCP servers on the same LAN. With Dnsmasq, you can create some neat forwarding rules. Actually, with plain DHCP you can specify an upstream/next server on PXE requests. Let’s PXE boot the kernel on real hardware.

Main DHCP Server

With just the principal (home) router on the LAN running Tomato, the extra Dnsmasq options are as simple as:

U-Boot DHCP requests come in and the response contains the instruction to fetch ipxe.efi (iPXE binary) from the TFTP server at 192.168.1.118. When iPXE loads and makes a DHCP request, its request is tagged with “ipxe” so this time the DHCP server responds with the instruction to fetch menu.ipxe from the same TFTP server. This avoids an infinite loop of iPXE loading itself.

Docker TFTP and HTTP Servers

Docker comes in handy again. One of the reasons I opted away from NFS bootloading is because NFS is tied to the kernel so it is not possible to dockerize an NFS service independent of the host kernel. TFTP is good for minimal file transfers like the secondary bootloader iPXE, and we’ll need an HTTP server, so this can all be trivially rolled into a Docker Compose YAML file. Just for some oompf, I added a Syslog server to catch log messages from U-Boot and iPXE.