Automatic Bootloader Testing - Rémi Duraffort [Automated Testing Summit 2019]

Bootloaders are complex. They should be tested as well, but currently it’s mostly manual testing only.

Rémi gave an introduction to LAVA. The important thing is that LAVA is intended to test on actual hardware, and it splits the roles over several modules. The server does things like API, scheduling, storing logs, sending notifications. The disptacher deploys the software, controls power, gets logs, etc. There’s a lavacli tool to talk to the server. A job definition YAML file has a device type, timeouts, priority, and a set of actions. Deploy action defines how to get the software to the device. Boot action detects when the boot has finishes and logs in. The tests are pulled from git repos and are shell scripts. They are copied into the filesystem image before deploy.

Some of the assumptions from LAVA don’t apply any more for bootloaders:

  • Known filesystem type (ext4, fat) in which it is possible to inject tests.
  • Provisioning possible with tftp, fastboot, …
  • Stable identifier e.g. MAC address, serial number - this is essential for the dispatcher. But in the bootloader it’s often not available yet.
  • Recovery method: normally the bootloader is used for recovery. When testing the bootloader and it crashes, there is no way to roll back or it is a manual process.
  • POSIX shell: bootloaders don’t typically have that.

For recovery, a possibility is SDMux, the SD card multiplexer. It makes it possible to switch the SD cards host between the DUT and the test system. This gives the test system full control over the SD card layout, it’s a simple setup. It’s open hardware so it can be modified as needed. It is (now) also quite reliable. Unfortunately, it adds some latency and some boards can’t handle that. A solution is to use a slow SD card.

An alternative way of provisioning is if there are two boot media controlled with a GPIO. Put the stable (known-working) bootloader in on of them, and the SUT in the otehr boot media. E.g. known-working in NAND and SUT in NOR. The dispatcher then first boots from NAND, in the boot loader it writes the SUT to NOR, then reset and boot from NOR, then run the tests. Of course, it still can be broken if the user flashes the NAND.

Since there is no shell, the normal approach of shell scripts can’t be used. LAVA fortunately also supports “interactive testing”, which is basically an expect script. No need to install anything on the DUT. However, it doesn’t support loops or conditions, so only a sequential test script is possible. It’s also really verboe. The interactive test is described directly in the configuration YAML file.

LAVA also supports to combine the bootloader and kernel tests. In the deploy stage, the kernel is deployed. In the test stage, the bootloader is loaded with an interactive script. Finally, there is a test that loads the zImage which was deployed before, and this triggers running the normal tests.

Currently the interactive script is part of the overall test description, which makes it very long. It would be better to include it from a separate repo (like is done for the shell scripts). Also, it would be good to allow to run a custom (expect) script on the disptacher, e.g. in a container, that can interact with the serial port that is occupied by the LAVA dispatcher.

A comment from the audience was that U-Boot already has a testing framework, which is a lot more extended than the tests that are specified in the LAVA file. Otherwise, we’re again creating more fragmentation. Also it must be possible to run negative tests, i.e. failure scenarios. Also with the tests inside the U-Boot source, we can make sure that they stay consistent, e.g. if an error message changes the test will be updated as well. The essential step here is for the LAVA dispatcher to make it possible to give the console access to something else (i.e. the U-Boot test Python scripts).

For provisioning, it is possible to use the “flasher” action. This is used e.g. on STM32 boards to flash over USB.