Before and after DMA operations we need memory barriers to ensure
stores on the CPU are observed by the device, and stores by the device
are observed by the CPU.
Previously we used a "dsb sy" instruction, however this is too strong
and can lead to slower performance.
Drew Gallatin found and tested the issue with cache-coherent devices,
reducing the barriers to dmb instructions in the outer-shareable
domain. After discussing the issue with the Arm Linux DMA maintainer I
concluded it was safe for non-coherent devices as the only dsb
instructions we need are after cache operations and these already exist.
Replace the dsb before DMA with a "dmb oshst" when cache-coherent, or
the existing dsb after invalidating the cache when not coherent. This
ensures earlier stores have completed before we signal the device.
Replace the dsb after DMA with "dmb oshld" in both cases. This ensures
later loads or stores are ordered after earlier device register loads.
Co-Authored-By: Andrew Gallatin <gallatin@FreeBSD.org>
Sponsored by: Arm Ltd
Sponsored by: Netflix