Jan-Simon talks about a few test systems that AGL has looked at, what they are using and what they might use in the future. Slides are available (PDF)
ptest from Yocto is a way to package the
make test or equivalent so it can be executed on the target. It runs the test and gives a
single PASS/FAIL/SKIP message. It also has a convention to run the test: the package should install a
run-ptest on the target, and the target will have an application
ptest-runner that goes through all the installed ptests. The ptest rules then also have some additional logic to fix up the built tests, e.g. replace build paths with target
paths. To use ptests, they have to be enabled in the distro and image features. ptest is well integrated in the bitbake cross-compilation and image generation process, including
running on qemu. However, it takes a lot of time to run all tests, and they generate a lot of output. It is hard to visualize that test log.
Fuego automates testing of an embedded target with a set of about 100 pre-packaged tests, in a Jenkins runner. It tests a production image, it cross-compiles additional executables for it (including LTP but also others) and uploads an runs them on the target. It grabs the results, parses and visualizes them, e.g. in graphs of benchmarks. It’s nice to have this large number of tests and visualization of the results. Requiring SSH access is a bit of a limitation. It also doesn’t include the deployment of the image that needs to be tested, so you should have a separate Jenkins job that deploys and make the Fuego test follow it. It also doesn’t support parallel tests, i.e. all tests run on a single board.
LAVA is a framework to automate deploying an image and executing a test. It manages the devices in a board farm, schedules tests on them and deploys and runs the test. It includes powercycling the board. It has templates for >= 150 boards. It supports the DUT to be remote from the runner, i.e. the LAVA master and the labs with the boards can be in different countries. It is very scalable. However, initial setup is complicated (improved with lava-docker image). Visualization is not as good as in Fuego.
KernelCI is a way to collect test results from hundreds of deployments. It started as aggregation of boot test results for the kernel. It is a database storing the results as JSON documents, and a webfrontend for visualization. The tests themselves can use any tool (e.g. Lava), kernelCI just aggregates. It is focused on boot tests but further tests are being added. Setup is a bit painful but there’s a kernelci-docker. Tailing the UI for different use cases is difficult though, but that is being redesigned.
labgrid takes care of managing boards. It takes control over power and serial, allows to reserve the boards, and exposes them to either a developer or a test tool. It abstracts the hardware specifics, which makes it easier to configure a board for e.g. Lava or Fuego. Setting it up is not too complicated.
r4d is a slightly similar use case. It was started by the
PREEMPT_RT developers. It also takes control over power and serial. It plugs into
libvirt, because Jenkins already has a libvirt-slave-plugin, so it is easy to connect a board to a Jenkins as an executor and run tests from Jenkins. It currently only supports a
limited set of power switches and serial servers. It also requires a patched libvirt, but they offer Debian packages.
There is a test stack survey on elinux.org that compares a lot of options.