Cluster Computer Gotchas: U-Boot, Device Trees, iPXE

The Pine64 clusterboard with seven SOPINEs (and 28 ARM64 cores) is wonderful. It does, however, have hardware that is dissimilar to the baseboard holding a single SOPINE. There is no HDMI nor audio output for one, and a few unnecessary (for cluster computing) IO ports are omitted such as the display panel and camera (MIPI) headers. There is a handy LED on GPIO pin 2 (PL7) of the 10×2 respective headers. There is also a Realtek RTL8211E with reported timing issues we have to deal with.

U-Boot

1. Allwinner EMAC TX Timing
2. Ethernet Voltage Ramp-Up
3. Realtek RTL8211E Timing Fix
4. Ethernet GMAC TX Delay
5. Spurious Timing Issue
6. Disable USB in U-Boot for Fast Booting
7. Disabling Distribution Defaults
8. SOPINE-on-Clusterboard Device Tree
9. Clusterboard LEDs as Boot Stage Indicators
10. Custom U-Boot Board for SOPINEs
11. U-Boot SPL and Full Builds
12. Clusterboard LEDs and Boot Progress Indicators (code)
13. Clusterboard LEDs Demonstration
14. Patch the SUNXI Watchdog Timer
15. Device Trees from Scratch
16. Error “.rodata will not fit in .sram”
17. SPI NOR-Flash Booting
18. Summary of Changed U-Boot Files

iPXE

1. Disable Interrupts in iPXE
2. Add More Menu Items
3. Disable iPXE USB
4. Speed up the iPXE Watchdog Reset

U-Boot on Pine64 Clusterboard Gotchas

U-Boot is a ubiquitous ARM64 bootloader, and this experience has let me dive deep into the C++ code and make helpful modifications. With U-Boot, it is possible to chain load iPXE to boot bare-metal hardware over HTTP(S). Very useful.

But first, here are some of the U-Boot gotchas the community solved but not yet in mainline U-Boot as of v2021.04. My hope is this information being in one place is helpful.

Allwinner EMAC TX Timing

A user (dippywood) in October 2020 in an Armbian forum came up with a script to patch a TX timing issue in the device tree for the SOPINE A64, the crux of which is increasing the timing delay from default 0ms to 500ms. As of 2021, this update is not in mainline U-Boot, so here is the procedure I followed.

The default TX timing of the EMAC is 0ms as seen below.

In the baseboard DTS, which is a surrogate for the clusterboard, there is no TX timing information: the default is then zero.

Searching other board DTS files, and the patch in the forum, it suffices to add one (or two) lines to the sun50i-a64-sopine-baseboard.dts and rename it to sun50i-a64-sopine-clusterboard.dts.

Going forward, let’s use sun50i-a64-sopine-clusterboard.dts as a new DTS file dedicated to the clusterboard.

Ethernet Voltage Ramp-Up

I’ll give the Ethernet PHY more time for the VCC to ramp up, say 300ms, from 100ms just in case that is an issue. Why not?

Realtek RTL8211E Timing Fix

In 2021, in the Mainline U-Boot sopine_baseboard_defconfig there is still no Realtek timing fix explicitly enabled. Actually, Realtek isn’t enabled at all, yet in March 2018 a commit added a RealTek RTL8211E timing fix. See below.

Let’s enable the RTL8211E in the clusterboard.

Now the clusterboard will be able to communicate with the onboard switch and packets will reach the network.

Ethernet GMAC TX Delay

From Linux Suni,

For reliable Gigabit networking (1000Mbit operation), several sunxi devices require an important tweak that adjusts the relative timing of the clock and data signals to the PHY, in order to compensate for differing trace lengths on the PCB … Recent mainline U-Boot uses CONFIG_GMAC_TX_DELAY to initialize these devices accordingly. If a necessary GMAC TX delay isn’t set, then GBit Ethernet operation might be unreliable or won’t work at all.

For completeness, let’s explore this more.

After searching the U-Boot repo for CONFIG_GMAC_TX_DELAY, let’s add CONFIG_GMAC_TX_DELAY=0 to the sopine_clusterboard_defconfig because visually I can see there is an effort to make the trace lengths the same on the clusterboard. Nice.

Spurious Timing Issue

It has been found that the generic ARMv8 timer has spurious timeouts. This tweak purports to fix spurious timeouts which have been seen during testing the SPI driver. Timeouts disappear when the number of bits are reduced to 10.

Let’s not forget to enable the erratum in the defconfig:

After flashing several SOPINES I’ve yet to notice any deleterious effects.

Disable USB in U-Boot for Fast Booting

Disabling USB in U-Boot to shave off 10s of start-up time may result in BOOTP/DHCP packets not leaving the hardware. There is a MAC address, but no network egress of packets. For example,

After the previous timing fixes, DHCP works consistently.

Disabling Distribution Defaults

Add the following to your sopine_clusterboard_defconfig to “manually” add back distro defaults without USB support.

SOPINE-on-Clusterboard Device Tree

There is no DTB file for the clusterboard. Let’s replace the contents in U-Boot’s arch/arm/dts/Makefile with just the essentials, noting to add the new clusterboard DTB (soon).

Customizing a device tree with fragments and aliases is beyond this article, but we can start by copying the sopine baseboard DTS, renaming the model, removing the missing hardware, and including inherited boards like below.

We should be able to use sun50i-a64-sopine-clusterboard.dtb with the Linux distro as well.

Clusterboard LEDs as Boot Stage Indicators

As alluded to earlier, there are seven LEDs – one for each SOPINE – on GPIO pin 2 (PL7) of each of the 10×2 GPIO headers. Let’s make that work for us during boot.

First, the DTS file is missing LED info. Let’s borrow some code from Linus Torvalds that is not in mainline U-Boot:

An easy way to blink a given LED during boot is invoke the following commands in a boot script (boot.scr):

This script can be expanded, but only runs during the execution of that boot script, not throughout the boot process.

Can we get more creative? Absolutely. Let’s take advantage of some built-in U-Boot facilities to make the LED blink at different boot stages. We could use the LED API, but if you are like me you will find it unsatisfying. Let’s be more sophisticated with the LED we have per SOPINE.

Where to begin? Here is a breadcrumb I found at the bottom of common/init/board_init.c:

Searching through the U-Boot source will yield the Kconfig for CONFIG_SHOW_BOOT_PROGRESS revealing a series of boot stage integers we can use to blink an LED or more.

For the curious, sprinkled throughout U-Boot are bootstage markers like the following:

Now, all we have to do (but don’t do this yet) is add a new method to board/sunxi/board.c like so:

We can see the boot progress numbers appear. Very nice.

What about making the clusterboard LEDs blink? We’ll come to that soon, but first, we need to talk more about hardware, next.

Custom U-Boot Board for SOPINEs

Do we really want to hack the common SUNXI board file to add LED support? Let’s complement it instead with a dedicated board for the clusterboard:

Because I like to write portable code, I’ve added TARGET_SOPINE_CLUSTERBOARD to Kconfig.

Now we have a dedicated menu entry to select SOPINEs on the clusterboard.

And then, in our new sopine-clusterboard.c file, we can write this stub:

But what about the LEDs? I’ll get to that very shortly. Proper setup is crucial.

U-Boot SPL and Full Builds

If we wanted to get fancy when using SPL U-Boot, we can differentiate between the SPL U-Boot build and the full U-Boot with CONFIG_SPL_BUILD. Why? U-Boot has flags to differentiate between regular builds and SPL builds, for example:

Whoops. Not realizing this, a code block using DM (Driver Model) intended to run in both boot stages may fail in the initial SPL stage in strange ways.

Clusterboard LEDs and Boot Progress Indicators

We have enough knowledge to blink the status LEDs at various bootstages from a very early stage, but not time zero. My sopine-clusterboard.c file now looks like the following:

From now on, a single LED blink means some DHCP network request. Two blinks mean autoboot is starting. Six blinks mean kernel handoff from U-Boot has started. This last stage means, for me, that the iPXE secondary bootloader has started (which needs its own blinking logic for more LED status indications). This is a big win.

Demo

Patch the SUNXI Watchdog Timer

In mainline U-Boot as of 2021.04 there is no SUNXI support for a watchdog timer. Try enabling one in menuconfig and you may pull your hair out. Even patching the device tree with a watchdog section isn’t enough. We will need a community patch. To prove it to yourself, search in mainline for SUNXI_WDT_TIMEOUT. If you cannot find that string in the repo, please continue reading.

A simple fix for this problem is to enable the internal hardware watchdog timer that Pine64-based SoCs have. We can enable it in the defconfig:

Device Trees from Scratch

Device trees are a pretty neat concept. With them, we can describe our target hardware without recompiling the Linux kernel. We can also map GPIO pins, specify how to access LEDs, and state what network chip we are using. I borrowed a book from the library about device trees, but this device trees video was far more helpful.

Key device tree concepts:

• Device tree files (DTS) can be split into several files and are amalgamated by #include directives.
• The final DTS files are prefixed with .dts. Included files are prefixed with .dtsi.
• Included files are underlaid, not just inserted at the #include directive point.
• Including files are overlaid on top of DTSI files.
• Replace just some parts with labels, not with full re-definitions.
• We can use #define to replace hard-coded or magic numbers.
• A “cell” just means a 32-bit integer.
• Strings like #size-cells = <1>; are not comments.

How DTSI underlays work:

We can then easily disable HDMI features of the SoC (not present on the clusterboard) with labels without having to copy the entire hardware defintion. For example,

Error “.rodata will not fit in .sram”

Having spent hours trying to trim SPL code here and there only to shave off mere hundreds of bytes, the following is all too common:

Taking a trip into the 700-page Allwinner A64 user manual we find the memory section.

SRAM A1 is 32 KiB (0x10000 - 0x17FFF), but what’s this? SRAM C starts right after SRAM A1 (0x18000 - 0x3FFFF). From the Allwinner block diagram, both memory blocks can be accessed by the same bus.

Let’s see if we can just overflow into SRAM C.

Then we must increase the SRAM size in a tool script. I’ll use 64 KiB (0x10000).

Success. Unless I’ve gotten lucky, this works repeatedly.

SPI NOR-Flash Booting

SPI NOR-flash on each computer module holds U-Boot which can chainload iPXE to boot Linux over HTTP(S). In order to flash U-Boot to NOR flash, certain commands need to be enabled by default. SOPINE uses the Winbond NOR flash chip w25q128 with erase size 4 KiB (you will need padding!) and a total of 16 MiB. Here are the settings:

Then, we’ll need a boot.scr script. Here is the one I use with comments and LED status indication:

Summary of Changed U-Boot Files

Here is a summary of the changed or added files to U-Boot as of v2021.04. I’ve created an overlay folder so that these files can be copied into U-Boot source instead of making yet another U-Boot fork. Why? Many hours were spent trying to decipher community memebers’ hack branches and PRs into U-Boot. Instead, I’ll have these all in one place for ease of reference.

iPXE Gotchas

To boot over HTTP(S), iPXE can complement U-Boot to achieve this.

Disable Interrupts in iPXE

iPXE has a smaller code base, so instead of using overlay files, I create them via a Docekrfile. Here is a fix for the above problem which essentially NOPs the cpu_nap() method resulting in running furious while-no-input loops, which is fine because iPXE propceeds quickly to boot the kernel.

Add More Menu Items

iPXE can be souped up with more usefull shell commands. iPXE is not limited in size like U-Boot is, so let’s add more commands. Again, I use a Dockerfile to achieve this. Here is pretty much the kitchen sink of commands we can add in the ARM64 implementation.

Disable iPXE USB

As with U-Boot, I’d like to disable USB functionality in iPXE to speed up loading with a Dockerfile.

Speed up the iPXE Watchdog Reset

Let’s shorten the watchdog timeout from five minutes to just one minute. iPXE then has one minute to hand off execution control to the kernel. This is sufficient for the use case of treating each node like a Lambda. Again, using a Dockerfile, here is the procedure.

More gotchas coming…