Linux Kernel Debugging: Going Beyond Printk Messages - Sergio Prado, Embedded Labworks [Open Source Summit EU 2019]

This talk is not about printk or static analysis tools, but about kernel debugging without printk.

To debug something you have to understand the problem, reproduce it, identify the source, and fix it. This talk if focused on step 3, identifying the source.

The top 5 problems are crashes, lockups, logic errors, resource leak, performance.

The first tool to use is your brain: you have to understand how the kernel works. The second tool is post-mortem analysis: logs and crash reports. Then you can use tracing/profiling. Printk debugging falls under this. You can do interactive debugging, i.e. gdb. Finally, there are debugging frameworks, that look at specific aspects, e.g. valgrind.

Not all tools are suitable for all jobs. For example, logs are generally not useful for debugging performance problems.

Kernel crash gives a stack trace. This gives us the last function that was executed (based on PC). If compiled with KALLSYMS it can resolve the address to a function + offset. If not, addr2line can be used to find the corresponding source line - you need the vmlinux ELF binary for that. But the kernel contains scripts/faddr2line that does the same based on the function + offset that you find in the kernel trace. Or you can load vmlinux in gdb with list(*addr) or list(function+offset)

If you don’t have a kernel console, you can use pstore (CONFIG_PSTORE) to save the printk buffer somewhere. This can be in memory (CONFIG_PSTORE_RAM) - then it can be retrieved after a soft reboot, e.g. reboot-on-panic. This needs a ramoops reserveration in the device tree. pstore has to be mounted on /sys/fs/pstore. If anything is stored in there, it will appear as files in that directory.

kdump is a different approach where you run a different kernel on crash. It’s like using kexec to reboot. Sergio had some problems getting it working for ARM, but in the end he managed. You need to give the crashkernel=NNM cmdline option to reserve space for the crash kernel (it shouldn’t use any of the memory that contains the crash info). Then you have to preload the crash kernel with kexec -p. Inside the crash kernel you have /proc/vmcore which is the core dump of the just-crashed kernel. There’s a kernel tool called crash that can be used to debug it. You can copy it to your laptop and load it into gdb.

To debug a running kernel with gdb, you need to enable kgdb in the kernel config. You need a serial port to connect the host running gdb itself with the kernel to be debugged. Network is also possible but needs patches. To put the kernel in debug mode, you need either a cmdline argument or set the kgdboc.kgdboc kernel parameter to the port, then use the G sysrq-trigger. To be able to use the serial port as a console simultaneously, use agent-proxy. Then you can connect to the console with a localhost telnet. The proxy will demultiplex the gdbserver packets out of the serial connection. When gdbserver is connected, it stops in kgdb_breakpoint. Also when the kernel crashes, it will stop in gdb at the crash point.

Another tool is tracing. There are two types of tracing: static and dynamic. Static tracing uses predefined trace points. Dynamic tracing is enabled at runtime. They are created using GCC’s profiling infrastructure. These tracing tools are used by higher-level tools like ftrace, trace-cmd, kernelshark, systemtap, perf, and even kernel live patching.

trace-cmd is a wrapper around ftrace. It enables and disables traces as needed, and logs the results in a file. The trace.dat file can be copied to the laptop and visualized in kernelshark.