Cluster Computer Gotchas: U-Boot, Device Trees, iPXE, Watchdogs

Goal: Locate in one place all the hardware and software gotchas I’ve encountered from compiling and running the U-Boot bootloader to fixing timing issues in a cluster computer of Allwinner A64 SoCs in order to help others and remember myself.

The Pine64 clusterboard with seven SOPINEs (and 28 ARM64 cores) is wonderful. It does, however, have hardware that is dissimilar to the baseboard holding a single SOPINE. There is no HDMI nor audio output for one, and a few unnecessary (for cluster computing) IO ports are omitted such as the display panel and camera (MIPI) headers. There is a handy LED on GPIO pin 2 (PL7) of the 10×2 respective headers. There is also a Realtek RTL8211E with reported timing issues we have to deal with.

Cluster computer gotchas


U-Boot

  1. Allwinner EMAC TX Timing
  2. Ethernet Voltage Ramp-Up
  3. Realtek RTL8211E Timing Fix
  4. Ethernet GMAC TX Delay
  5. Spurious Timing Issue
  6. Disable USB in U-Boot for Fast Booting
  7. Disabling Distribution Defaults
  8. SOPINE-on-Clusterboard Device Tree
  9. Clusterboard LEDs as Boot Stage Indicators
  10. Custom U-Boot Board for SOPINEs
  11. U-Boot SPL and Full Builds
  12. Clusterboard LEDs and Boot Progress Indicators (code)
  13. Clusterboard LEDs Demonstration
  14. Patch the SUNXI Watchdog Timer
  15. Device Trees from Scratch
  16. Error “.rodata will not fit in .sram”
  17. SPI NOR-Flash Booting
  18. Summary of Changed U-Boot Files

iPXE

  1. Disable Interrupts in iPXE
  2. Add More Menu Items
  3. Disable iPXE USB
  4. Speed up the iPXE Watchdog Reset

U-Boot on Pine64 Clusterboard Gotchas

U-Boot is a ubiquitous ARM64 bootloader, and this experience has let me dive deep into the C++ code and make helpful modifications. With U-Boot, it is possible to chain load iPXE to boot bare-metal hardware over HTTP(S). Very useful.

But first, here are some of the U-Boot gotchas the community solved but not yet in mainline U-Boot as of v2021.04. My hope is this information being in one place is helpful.

Allwinner EMAC TX Timing

Gotcha: There is an Ethernet timing-issue fix that needs to be manually applied to U-Boot.

A user (dippywood) in October 2020 in an Armbian forum came up with a script to patch a TX timing issue in the device tree for the SOPINE A64, the crux of which is increasing the timing delay from default 0ms to 500ms. As of 2021, this update is not in mainline U-Boot, so here is the procedure I followed.

The default TX timing of the EMAC is 0ms as seen below.

In the baseboard DTS, which is a surrogate for the clusterboard, there is no TX timing information: the default is then zero.

Searching other board DTS files, and the patch in the forum, it suffices to add one (or two) lines to the sun50i-a64-sopine-baseboard.dts and rename it to sun50i-a64-sopine-clusterboard.dts.

TODO: Are phy-supply = <&reg_gmac_3v3>; and phy-mode = "rgmii-id"; better, as used in the Rockchip DTS? DM me if you find out.

Going forward, let’s use sun50i-a64-sopine-clusterboard.dts as a new DTS file dedicated to the clusterboard.


Ethernet Voltage Ramp-Up

Gotcha: Give more time to the Ethernet chips to reach VCC before beginning networking.

I’ll give the Ethernet PHY more time for the VCC to ramp up, say 300ms, from 100ms just in case that is an issue. Why not?


Realtek RTL8211E Timing Fix

Gotcha: The clusterboard Ethernet RTL8211E chips need another timing fix enabled, and turned on.

In 2021, in the Mainline U-Boot sopine_baseboard_defconfig there is still no Realtek timing fix explicitly enabled. Actually, Realtek isn’t enabled at all, yet in March 2018 a commit added a RealTek RTL8211E timing fix. See below.

Let’s enable the RTL8211E in the clusterboard.

Now the clusterboard will be able to communicate with the onboard switch and packets will reach the network.


Ethernet GMAC TX Delay

Gotcha: There may need to be timing compensation for PCB Ethernet TX and RX trace lengths not being equal.

From Linux Suni,

For reliable Gigabit networking (1000Mbit operation), several sunxi devices require an important tweak that adjusts the relative timing of the clock and data signals to the PHY, in order to compensate for differing trace lengths on the PCB … Recent mainline U-Boot uses CONFIG_GMAC_TX_DELAY to initialize these devices accordingly. If a necessary GMAC TX delay isn’t set, then GBit Ethernet operation might be unreliable or won’t work at all.

For completeness, let’s explore this more.

After searching the U-Boot repo for CONFIG_GMAC_TX_DELAY, let’s add CONFIG_GMAC_TX_DELAY=0 to the sopine_clusterboard_defconfig because visually I can see there is an effort to make the trace lengths the same on the clusterboard. Nice.


Spurious Timing Issue

Gotcha: Compensate for a timer glitch that may cause spurious timeouts in U-Boot.

It has been found that the generic ARMv8 timer has spurious timeouts. This tweak purports to fix spurious timeouts which have been seen during testing the SPI driver. Timeouts disappear when the number of bits are reduced to 10.

Let’s not forget to enable the erratum in the defconfig:

After flashing several SOPINES I’ve yet to notice any deleterious effects.


Disable USB in U-Boot for Fast Booting

Gotcha: Disabling DISTRO_DEFAULTS in U-Boot may break everything.

Disabling USB in U-Boot to shave off 10s of start-up time may result in BOOTP/DHCP packets not leaving the hardware. There is a MAC address, but no network egress of packets. For example,

After the previous timing fixes, DHCP works consistently.


Disabling Distribution Defaults

Gotcha: If you disable CONFIG_DISTRO_DEFAULTS in your defconfig file to remove USB support, be sure to manually add back the various boot commands it formerly selected. Without this, our SPI flashing script will not run.
U-Boot disable distro defaults
U-Boot disable distro defaults

Add the following to your sopine_clusterboard_defconfig to “manually” add back distro defaults without USB support.


SOPINE-on-Clusterboard Device Tree

Gotcha: Clusterboard SOPINEs do not have a dedicated device tree.

There is no DTB file for the clusterboard. Let’s replace the contents in U-Boot’s arch/arm/dts/Makefile with just the essentials, noting to add the new clusterboard DTB (soon).

Customizing a device tree with fragments and aliases is beyond this article, but we can start by copying the sopine baseboard DTS, renaming the model, removing the missing hardware, and including inherited boards like below.

We should be able to use sun50i-a64-sopine-clusterboard.dtb with the Linux distro as well.


Clusterboard LEDs as Boot Stage Indicators

Gotcha: It is not well documented how to access the status LEDs on the clusterboard from U-Boot.

As alluded to earlier, there are seven LEDs – one for each SOPINE – on GPIO pin 2 (PL7) of each of the 10×2 GPIO headers. Let’s make that work for us during boot.

First, the DTS file is missing LED info. Let’s borrow some code from Linus Torvalds that is not in mainline U-Boot:

An easy way to blink a given LED during boot is invoke the following commands in a boot script (boot.scr):

This script can be expanded, but only runs during the execution of that boot script, not throughout the boot process.

Can we get more creative? Absolutely. Let’s take advantage of some built-in U-Boot facilities to make the LED blink at different boot stages. We could use the LED API, but if you are like me you will find it unsatisfying. Let’s be more sophisticated with the LED we have per SOPINE.

Where to begin? Here is a breadcrumb I found at the bottom of common/init/board_init.c:

Searching through the U-Boot source will yield the Kconfig for CONFIG_SHOW_BOOT_PROGRESS revealing a series of boot stage integers we can use to blink an LED or more.

For the curious, sprinkled throughout U-Boot are bootstage markers like the following:

Gotcha: Wait a moment, what is that constant just above? Use instead bootstage.h to see the real integers passed to show_boot_progress().

Now, all we have to do (but don’t do this yet) is add a new method to board/sunxi/board.c like so:

We can see the boot progress numbers appear. Very nice.

What about making the clusterboard LEDs blink? We’ll come to that soon, but first, we need to talk more about hardware, next.


Custom U-Boot Board for SOPINEs

Gotcha: There is no board (anymore?) for the Pine64 family, just the common SUNXI board.

Do we really want to hack the common SUNXI board file to add LED support? Let’s complement it instead with a dedicated board for the clusterboard:

Because I like to write portable code, I’ve added TARGET_SOPINE_CLUSTERBOARD to Kconfig.

Now we have a dedicated menu entry to select SOPINEs on the clusterboard.

New U-Boot menu item for SOPINEs
New U-Boot menu item for SOPINEs

And then, in our new sopine-clusterboard.c file, we can write this stub:

But what about the LEDs? I’ll get to that very shortly. Proper setup is crucial.


U-Boot SPL and Full Builds

Gotcha: The same code is compiled for SPL U-Boot (under 32-KiB bootloader), and the full U-Boot code, but may run differently causing many headaches.

If we wanted to get fancy when using SPL U-Boot, we can differentiate between the SPL U-Boot build and the full U-Boot with CONFIG_SPL_BUILD. Why? U-Boot has flags to differentiate between regular builds and SPL builds, for example:

Whoops. Not realizing this, a code block using DM (Driver Model) intended to run in both boot stages may fail in the initial SPL stage in strange ways.


Clusterboard LEDs and Boot Progress Indicators

Gotcha: Clusterboard status GPIOs and LEDs cannot be activated the microsecond power is applied.

We have enough knowledge to blink the status LEDs at various bootstages from a very early stage, but not time zero. My sopine-clusterboard.c file now looks like the following:

From now on, a single LED blink means some DHCP network request. Two blinks mean autoboot is starting. Six blinks mean kernel handoff from U-Boot has started. This last stage means, for me, that the iPXE secondary bootloader has started (which needs its own blinking logic for more LED status indications). This is a big win.

Demo

Request for advice: If you know a better way to write to GPIO memory/registers directly to avoid waiting for the DM initialization, please reach out to me. My solution above is the result of timeboxing.

Patch the SUNXI Watchdog Timer

Gotcha: If U-Boot or iPXE fails, you’ll be dropped to the U-Boot shell forever. Patch and enable the internal hardware watchdog reset timer.

In mainline U-Boot as of 2021.04 there is no SUNXI support for a watchdog timer. Try enabling one in menuconfig and you may pull your hair out. Even patching the device tree with a watchdog section isn’t enough. We will need a community patch. To prove it to yourself, search in mainline for SUNXI_WDT_TIMEOUT. If you cannot find that string in the repo, please continue reading.

A simple fix for this problem is to enable the internal hardware watchdog timer that Pine64-based SoCs have. We can enable it in the defconfig:


Device Trees from Scratch

Gotcha: We cannot just pick and choose parts of another device tree to cobble together for a new board from community edits and hacks. Let’s be methodical and take the time to learn how device trees work.

Device trees are a pretty neat concept. With them, we can describe our target hardware without recompiling the Linux kernel. We can also map GPIO pins, specify how to access LEDs, and state what network chip we are using. I borrowed a book from the library about device trees, but this device trees video was far more helpful.

Device tree concepts
Device tree concepts (source: Bootlin)
Tip: The canonical DTS files for various SoC boards are found in the Linux repo, not the U-Boot repo. Start with the Linux repo when including or customizing board features.

Key device tree concepts:

  • Device tree files (DTS) can be split into several files and are amalgamated by #include directives.
  • The final DTS files are prefixed with .dts. Included files are prefixed with .dtsi.
  • Included files are underlaid, not just inserted at the #include directive point.
  • Including files are overlaid on top of DTSI files.
  • Replace just some parts with labels, not with full re-definitions.
  • We can use #define to replace hard-coded or magic numbers.
  • A “cell” just means a 32-bit integer.
  • Strings like #size-cells = <1>; are not comments.

How DTSI underlays work:

DTSI overlays when included in DTS files
DTSI overlays when included in DTS files (source: Bootlin)

We can then easily disable HDMI features of the SoC (not present on the clusterboard) with labels without having to copy the entire hardware defintion. For example,


Error “.rodata will not fit in .sram”

Gotcha: The 32 KiB limit of U-Boot SPL is too limiting, especially if we enable logging. Let’s bump that up.

Having spent hours trying to trim SPL code here and there only to shave off mere hundreds of bytes, the following is all too common:

Taking a trip into the 700-page Allwinner A64 user manual we find the memory section.

Allwinner A64 memory map
Allwinner A64 memory map

SRAM A1 is 32 KiB (0x10000 - 0x17FFF), but what’s this? SRAM C starts right after SRAM A1 (0x18000 - 0x3FFFF). From the Allwinner block diagram, both memory blocks can be accessed by the same bus.

Allwinner A64 block diagram and SDRAM
Allwinner A64 block diagram and SDRAM

Let’s see if we can just overflow into SRAM C.

Then we must increase the SRAM size in a tool script. I’ll use 64 KiB (0x10000).

Success. Unless I’ve gotten lucky, this works repeatedly.


SPI NOR-Flash Booting

Gotcha: Enable the SF command and Winbond driver to flash U-Boot to NOR flash for bare-metal booting.

SPI NOR-flash on each computer module holds U-Boot which can chainload iPXE to boot Linux over HTTP(S). In order to flash U-Boot to NOR flash, certain commands need to be enabled by default. SOPINE uses the Winbond NOR flash chip w25q128 with erase size 4 KiB (you will need padding!) and a total of 16 MiB. Here are the settings:

Then, we’ll need a boot.scr script. Here is the one I use with comments and LED status indication:


Summary of Changed U-Boot Files

Here is a summary of the changed or added files to U-Boot as of v2021.04. I’ve created an overlay folder so that these files can be copied into U-Boot source instead of making yet another U-Boot fork. Why? Many hours were spent trying to decipher community memebers’ hack branches and PRs into U-Boot. Instead, I’ll have these all in one place for ease of reference.


iPXE Gotchas

To boot over HTTP(S), iPXE can complement U-Boot to achieve this.

Disable Interrupts in iPXE

Gotcha: U-Boot (UEFI emulation) doesn’t support interrupts as it is single-threaded, and iPXE needs to be woken up on input.

iPXE has a smaller code base, so instead of using overlay files, I create them via a Docekrfile. Here is a fix for the above problem which essentially NOPs the cpu_nap() method resulting in running furious while-no-input loops, which is fine because iPXE propceeds quickly to boot the kernel.

Add More Menu Items

Gotcha: iPXE has many more network tools available, but they are disabled by default.

iPXE can be souped up with more usefull shell commands. iPXE is not limited in size like U-Boot is, so let’s add more commands. Again, I use a Dockerfile to achieve this. Here is pretty much the kitchen sink of commands we can add in the ARM64 implementation.


Disable iPXE USB

Gotcha: We don’t need USB support in iPXE as it is purely for network booting.

As with U-Boot, I’d like to disable USB functionality in iPXE to speed up loading with a Dockerfile.


Speed up the iPXE Watchdog Reset

Gotcha: When iPXE fails for some reason, the watchdog reset will take place in 5 minutes. Not knowing this, one would think a cluster node is hung.

Let’s shorten the watchdog timeout from five minutes to just one minute. iPXE then has one minute to hand off execution control to the kernel. This is sufficient for the use case of treating each node like a Lambda. Again, using a Dockerfile, here is the procedure.


More gotchas coming…