use the BUS_DMA_NOWRITE flag to expose and create the read-only VT-d IOMMU mappings

[tychon]

Summary

For drivers which want to protect DMA source operands against spurious accesses the VT-d IOMMU in re-map mode can be used to support read-only mappings. Use the BUS_DMA_NOWRITE flag to expose and create the TTEs for those mappings.

[kib]

The flag is only supported by sparc64, and there is no single use of it in the tree.

[tychon]

In D19729#422689, @kib wrote:

The flag is only supported by sparc64, and there is no single use of it in the tree.

Indeed there is no current in tree user. It is however used outside of the tree in conjunction with a Mellanox Innova-2 card. Plus, once in tree anyone is free to use and it will no longer be sparc64 only :-)

[kib]

In D19729#422726, @tychon wrote:

In D19729#422689, @kib wrote:

The flag is only supported by sparc64, and there is no single use of it in the tree.

Indeed there is no current in tree user. It is however used outside of the tree in conjunction with a Mellanox Innova-2 card. Plus, once in tree anyone is free to use and it will no longer be sparc64 only :-)

I am curious. Is this some private code to communicate with FPGA ? (Because I know both busdma dmar and in-tree ml5_fpga too closely).

[tychon]

In D19729#422742, @kib wrote:

In D19729#422726, @tychon wrote:

In D19729#422689, @kib wrote:

The flag is only supported by sparc64, and there is no single use of it in the tree.

Indeed there is no current in tree user. It is however used outside of the tree in conjunction with a Mellanox Innova-2 card. Plus, once in tree anyone is free to use and it will no longer be sparc64 only :-)

I am curious. Is this some private code to communicate with FPGA ? (Because I know both busdma dmar and in-tree ml5_fpga too closely).

Private code? Yes and no.

We have an FPGA image which provides a compression/decompression engine. The engine interfaces with the system by incorporating Xilinx PCI Express DMA/bridge IP (https://www.xilinx.com/support/documentation/ip_documentation/xdma/v4_1/pg195-pcie-dma.pdf and https://www.xilinx.com/support/documentation/ip_documentation/axi_pcie3/v3_0/pg194-axi-bridge-pcie-gen3.pdf). The IP instantiates a PCI Express device with BAR access to the engines CSRs and DMA access to the serial interfaces which feed the engines data path. There is a dual-licensed driver for the DMA component -- sources here (https://www.xilinx.com/support/answers/65444.html) if you are curious about that. It was ported by cem@ (who hopes to upstream it) and largely utilizes the linuxkpi. There is also a private companion driver to manage the engines CSRs.

Since the PCI Express endpoint and associated data paths are synthesized on the FPGA they have a larger than average vulnerability to SEUs. Hence we want to limit the "aperture" by which the DMA/bridge can scribble on memory if things go awry. That's done both by constraining access with the IOMMU and even with that limiting write-only access (using the BUS_DMA_NOWRITE) to bus_dmamap_load()) even further.

[kib]

In D19729#422803, @tychon wrote:

Since the PCI Express endpoint and associated data paths are synthesized on the FPGA they have a larger than average vulnerability to SEUs. Hence we want to limit the "aperture" by which the DMA/bridge can scribble on memory if things go awry. That's done both by constraining access with the IOMMU and even with that limiting write-only access (using the BUS_DMA_NOWRITE) to bus_dmamap_load()) even further.

Do you enable busdma DMAR for everything, or limit the scope to the fpga device only ? If the later, do you use hw.busdma.pciX.X.X.X tunables to control that, or have something more involved ?

[tychon]

In D19729#422870, @kib wrote:

In D19729#422803, @tychon wrote:

Since the PCI Express endpoint and associated data paths are synthesized on the FPGA they have a larger than average vulnerability to SEUs. Hence we want to limit the "aperture" by which the DMA/bridge can scribble on memory if things go awry. That's done both by constraining access with the IOMMU and even with that limiting write-only access (using the BUS_DMA_NOWRITE) to bus_dmamap_load()) even further.

Do you enable busdma DMAR for everything, or limit the scope to the fpga device only ? If the later, do you use hw.busdma.pciX.X.X.X tunables to control that, or have something more involved ?

Our goal is to have busdma DMAR enabled for all devices. There are still a few drivers I'm tweaking so I use the hw.busdma.pciX.X.X.X tunables to special case those for now. Seems to work nicely so no need for anything involved.