Keynote: The Kernel Report - Jonathan Corbet, Author, Kernel Developer and Executive Editor, [Open Source Summit EU 2018]

Jon gives an overview of recent evolutions in the kernel.

[Note: all this has also been discussed on]

7 major releases in the last 14 months. Every release contains about 14000 changesets. It’s getting pretty boring, it’s always business as usual. The only exception 4.15 which took 11 weeks instead of the usual 9-10, thanks to Meltdown/Spectre. These are hardware vulnerabilities - it looks a lot like software vulnerabilities, and we have a process for that. For meltdown/spectre, however, this process was not followed. Instead, there was a whole lot of secrecy - the kernel community was only informed in October, and then they were not allowed to talk to each other. The result was that every distribution shipped something different, of varying quality.

The fixes for Meltdown were developed in public (with an unconvincing cover story), and were in good shape by the time the vulnerability was disclosed. Distros used a fairly uniform solution.

Spectre fixes were done in private, and were not in shape by disclosure. Every distro shipped something different, nothing similar to upstream.

It also led to a lot of developer burnout and frustration - they put in a whole lot of effort with not so great results.

Because of all the secrecy, many distros were left in the cold, both in the open source community but also commercial cloud providers. Only the biggest players had access.

The good news is that the lessons have been learned. On the other hand, we have also learned that the hardware is not quite the solid foundation we thought it to be.

Most of us run a stable kernel rather than mainline. These get about 10000 fixes compared to mainline. However, compared to mainline, that’s only a little bit. E.g. 4.9 stable got less than 10000 fixes, but the mainline got 136000 changesets over the same time. So there is ongoing work to capture more of the fixes into stable.

There is a tendency for longer term support, e.g. 4.4 to 2022 and 4.9 to 2023, and CII even wants 10-20 years of support.

Unfortunately we also get regressions in stable, to the extent that distributors are having second thoughs about using -stable. Also, some of the fixes get very invasive (with meltdown/spectre as the highlight). As a result, not everything does get backported.

This makes people wonder if it really is a good idea to have this longterm stable illusion. In the end, nothing much is stable about it except for the version number. So it is probably better to continuously migrate to the latest stable instead of sticking in a longterm series. One little problem with that idea is ancient hardware.

One of the fast moving areas in the kernel is BPF. It’s an in-kernel VM with a verifier and a JIT compiler. It shows up into seccomp policy, protocol implementations (IR remote control), tracing, packet filtering (replacing existing firewall code), network processing (XDP, Express Data Path). So BPF is not just used to supplement kernel functionality, but also to replace some of it.

BPF is a way to push code into the kernel, but there’s also the reverse movement: XDP, seccomp traps, ELF modules, and even handling memory faults in userspace.

So we’re seeing the boundary between kernel and userspace becoming more porous and less clear.

The final point is conduct in the kernel community. In the good old days, there was no change tracking, no release discipline, no anti-regression rules, … and no code of conduct. It was not a professional environment, it was the wild west. Over the years, most of these things have been addressed and the community has become more professional. Most people agree that this was for the better. The final thing still missing was a code of conduct. This has been fixed as of last month. It does create some angst in the community, fearing that we will have to accept code that is not up to snuff or that we will have to release control over the kernel. Jon believes that it is going to pan out well, however, and that it will still be fun to work on the kernel.