Being Systematic with systemd
Chris Simmonds, 2net Ltd [Open Source Summit EU 2022]

There is some resistance to using systemd in embedded. Chris tries to show how it can be used effectively. He had a previous talk in 2019 covering boot speed, so that’s not covered here.

Bootstrapping is best expressed as a hierarchy, tree structure, of dependencies. The root of the tree is the target. The daemon walks down from the target to find what needs to be started to reach that target. Every node in the hierarchy is called a unit. Most are service units, but there are others as well.

All units have a [Unit] section. It has some generic information, but most importantly expresses dependencies: Requires, Wants, Conflicts. The difference between Requires and Wants is that for the latter, failure to start the dependency (because it doesn’t exist or it crashes or whatever) doesn’t cause the unit itself to fail. Reverse dependencies are expressed in the [Install] section rather than [Unit].

The dependencies don’t express ordering, they only determine which units need to be started. When a unit is activated, it is added to the activation queue, then its dependencies are added to the activation queue. The activation queue is however not ordered, systemd will just start things in any order it likes. To express order, use Before and After. This imposes a partial ordering on the activation queue.

Units are searched in a hardcoded search path: /etc/systemd/system is for local configuration, /usr/lib/systemd/system is for the distribution-wide configuration, /run/systemd/system if for dynamic things - mostly added by systemd itself. To override or remove a unit, just put a file in /ets/systemd/system with the same name as the one in /usr/lib/systemd/system. An empty file (or symlink to /dev/null) effectively disables the unit.

The [Service] section in a service unit specifies how the service should be started, stopped, what type it is, etc. A number of environment variables are set by systemd which can be used when specifying what to run.

Types of service:

  • simple is the default. The program is launched in the background. If it exits, it will be restarted (but if it keeps doing that, systemd gives up after a while).
  • oneshot runs it once, even if it exits.
  • forking is for when the service forks itself, i.e. if it uses daemonize() or something like that.

A target is a type of unit that ends in .target, but it doesn’t do anything itself. It’s just there for adding dependencies. The default target is the multi-user target, it’s set on the commandline with and aliases to it. It requires and conflicts with Typically the targets don’t depend on services, rather services have a reverse dependency on the target in the [Install] section. This has the effect of creating a symlink in /usr/lib/systemd/system/<target>.wants when the unit is installed or in /etc/systemd/system/<target>.wants with systemctl enable. This is typically with the WantedBy, not RequiredBy, so the target is still reached even if some service fails.

To debug the presence or absence of a service, you can use systemctl list-dependencies.

Services can be started up on demand with socket activation. This is controlled with a socket unit. There always has to be a matching service unit as well - with the same name or different by specifying Service=. A “socket” can be a Unix socket, a TCP socket, a POSIX mqueue, etc. etc. ListenSpecial allows to activate on a POLLIN event on a file.

A timer is similar to a trigger but time based instead of socket activated. This can be used for something like a cron job, or by delaying startup.

The Restart option specifies what needs to be done if the service fails. It has some controls for how many retries, to rate limit it, etc. You can also specify a separate cleanup service. You can even specify FailureAction=reboot. A watchdog-enabled service has to ping systemd on a regular basis with sd_notify - if it doesn’t, it is restarted. Similarly, systemd itself can feed the kernel watchdog.