Linux has a unique development model. Even though it is continuously improving, regressions do come in. Thus, CI is needed.
A large part of the kernel is drivers. The pace of change is really high. There are no architects, but there are rules: regressions get reverted, each feature needs to have a open source userspace user. This is what got Linux out of a niche into mainstream: it is strictly improving every version.
Still, regressions do come in. This triggers vendors to freeze on a specific version.
Why do they get in? It is a huge codebase with a lot of shared code. New version every 10 weeks. Developers typically just test on their own machine, there are no (unit) test suites. There are few self tests. Human-powered QA won’t help either, there are too many configurations and use cases, and the pace of change is too high. -rc is actually intended for this, but in practice not many people use -rc kernels.
Pre-merge testing allows putting the cost of integration on the person doing the development - they are anyway better placed for analysing the problem. It also helps to get a better global understanding to developers. But then, the test system must be fast, and public so everybody can do them. Also, integration is difficult, because merging in the Linus tree brings in thousands of changes. So filtering failures is needed.
What is needed in the test reports? Provide all necessary information: machine info, what was tested, full logs. Also every tested version should be tagged to make it easier to bisect, and also store the compiled versions.
Still, integration testing is very noisy: things fail all the time for various reasons, including failing hardware. Thus, known issues must be filtered. The tool should take the post-merge issues and match them to bugs based on signatures in the test results. Then developers can assign impact so they can focus on the most important bugs. Also automatic bisecting would be great.
Such a tool exists: CI Bug Log It generates a report out of each test result. To create a new filter, a web interface allows to select the machine (or machine tag), test, test result, and regex in the test output. In real time, it shows how many uncategorized failures are matched by it.
It generates graphs to track which bugs get hit most often over time.
Comparison to other kernel CI:
At XDC, a new community was started to make a distributed GFX CI similar to kernel CI, so it can also be done for other GFX than Intel. It will be an open source toolbox. The IGT test suite has a number of driver-agnostic tests that can be used for other drivers. Google has made chamelium to emulate plugging screens in and out, and for HDMI-CEC.