DMA Safety in Buffers for Linux Kernel Device Drivers - Wolfram Sang, Renesas / Consultant [Open Source Summit EU 2018]

Wolfgang learned about DMA safety the hard way when he made the i2c subsystem ready for using the DMA subsystem.

Usual use case for I2C is many small messages (to read or set a register). Therefore, DMA was never considered. Also there were no rules about how buffers should be allocated, so they come from the heap, the stack, rodate, kmapped memory, … To avoid needing to convert zillions of drivers, Wolfram has made DMA optional for I2C.

Ideally, it should be possible at runtime to discover if a buffer is dma capable. Unfortunately, there is no function is_dma_capable() - rightly so because it would be extremely difficult to do that check. So instead, there is an opt-in approach: I2C_M_DMA_SAFE in the message flags marks a DMA-safe buffer. It means that a lot of drivers need to be manually audited, but at least it is safe.

On the master side, the function i2c_get_dma_safe_msg_buf() checks that flag and either uses the buffer directly or copies into a bounce buffer. But also on the master side that’s optional, the master can also check the flag directly (e.g. to avoid needless copying, or to get the buffer from a pool rather than allocating).

Messages that come from userspace (i2c-dev) are always DMA safe, so they are marked as such. Emulated SMBus block transfers are also DMA safe. regmap is a heavy I2C user as well, so Wolfram checked what is there. Turns out that regmap just assumes that everything is DMA safe.

In fact, there is no generic way even to allocate a DMA-safe buffer, because it depends on the device that does the DMA - some devices have only a limited view of the physical memory. Therefore, dma_map_single will allocate a bounce buffer if needed (except on arm32, where it isn’t implemented yet).

This flag solution works, but only for I2C. It would really help a lot if there was some generic way to mark buffers as DMA-safe. Some other subsystems (SLIMBus, SPMI, OneWire) are in the same situation as I2C was, so they would also potentially benefit from such a flag.

The SPI subsystem requires DMA support, i.e. the buffers are required to be DMA-capable, and it’s documented as such.

Getting a DMA buffer right is difficult. For example, if it’s allocated as part of a struct, it’s probably not aligned. devm_kmalloc() puts the allocated data in a struct with something in front of it, so you can’t use that for DMA. Stack-allocated buffers do sometimes work, but not if e.g. somebody enables CONFIG_VMAP_STACK, or e.g. on ARM you will eventually get data corruption on your stack.

There is no good way for detecting wrong DMA allocations at runtime. Enabling CONFIG_DMA_API_DEBUG helps but it doesn’t always find the problems. If it does find something, fix it even things otherwise work for you. The problems created can be hard to find and subtle.

Therefore, it is essential to make clear rules as soon as possible, and keep an eye on it during review.