Beaker Project: Automated Testing at Red Hat - Tomas Klohna, Red Hat [Automated Testing Summit 2019]

RedHat uses Beaker for test automation since 2010. It is a unified testing platform for quality engineers.

Beaker has a hardware focus, compared to most other platforms. It targets multiple architectures, including powerpc and s390. As a side effect, it offers hardware filtering. When a machine is added to the Beaker lab, RHEL is installed on it and it is queried to see what hardware it contains. This is called the inventory management. That info is stored in a database, and is then used to schedule only tests that can work on that hardware. The database also tracks the machine history, e.g. when a disk was changed and who did it. This also tracks which tests were run, which is important if a test turns out to be destructive. It is also used for access control, to give ACLs for specific users to a specific machine.

Beaker tests the full stack, not just the kernel or a single userspace. So it tests installation, booting, and userspace components. The test results and logs are gathered and can be exposed in other tools, recently kcidb was added.

Tests are distributed over multiple machines according to the filtering and availability of the machines. This is the scheduling. A job consists of some recipe sets which consists of recipes. It is described in XML. It contains filters for the distro and for the host.

Provisioning is done with a lab controller. Once a job is scheduled to a specific machine, a PXE config for it is passed to the lab controller. The lab controller powers up the machine and it gets the PXE config and performs initial boot. Beaker assumes that it uses anaconda for installation, because that supports callbacks that report installation progress and when it is complete. Then the lab controller reboots the machine and testing can start.

Restraint is a somewhat separate component that performs the actual tests. It was developed last year. It is written in C, because it has to target RHEL5 and RHEL8 at the same time. Restraint sets up the task environment. The fail/warn/pass states are described and reported. It supports virtual guests when the beaker config describes that it has to run on a virtual guest. It also has watchdogs to terminate tasks. Sometimes, however, the machine has to be terminated externally by the lab controller.

It is a very complicated project, but still contributions are welcome. https://github.com/beaker-project and https://beaker-project.org