Defining a Standard Board Management API and Working Together to Build a Modular CI Ecosystem - Discussion Session [Automated Testing Summit 2019]

Even though we already have a proliferation of “standards” with the different test projects, we will need to define an additional one to be able to reuse things and collaborate better. The ultimate goal is to reduce our workload.

The last sessions were merged because everybody feels a need for more discussion and collaboration.

To be able to focus on only part of the flow, we have to standardise on how things are split up (i.e. what are the components) and what are the APIs between them. There’s a big figure which identifies 16 APIs and about 10 components. There are modules (which do something) and servers/repositories (where the data is stored, e.g. the test definitions, result artefacts).

kcidb is the results database, it contains URLs to the actual result artefacts.

One thing that would be nice is a low-end expect tool that can run on the target.

For power control, people converged on pudaemon.

If we want to combine the strengths from the different frameworks, we will need to break them up into pieces that are usable as components.

Tim proposes to converge at least on the style of interfacing: git-CLI-style, so <tool> <verb> <args>, with result in JSON (except for single-value results or bulk data like log files or tarballs). Also converge on the verbs: start, stop, collect. And async output goes to files. CLI API can easily wrap around a different IPC or REST API. In general, these calls are not time-critical so CLI is OK. Many systems already have a CLI but it’s not standardised.

We all have existing systems that we can’t afford to break. So you have to take it one little thing at a time, it will be a slow process. We should also look at each other’s systems and see what are the strengths. Ideally then it should be possible to grab that little piece without taking the entire system.

Problem is that there is no incentive to do this until you’ve done it. Someone has to go first. And there’s always the danger of locking into a bad interface.

We can start with small things that are done in a tool and make this available separately, e.g. seddiff, results parser.

A nice starting feature would be to be able to run different frameworks in one lab. However, the limitation here is the target control. Most frameworks have their specific target control. By unifying the target control API, it would be possible to use the same board with target control in different frameworks.