virtio: get a correct tag for IOMMU operation

[br]

Summary

Ensure that PCI bus BUS_GET_DMA_TAG() method sees the actual PCI device which makes the request for dma tag, instead of some descendant of the PCI device (e.g. virtio devices on a transport bus).

Also, accept the VIRTIO_F_IOMMU_PLATFORM feature which is present on IOMMU-enabled emulators.

This is needed for the virtio block device operation on a IOMMU-enabled system like ARM System MMU or RISC-V IOMMU.

This patch depends on these reviews (busdma support for virtqueue and virtio block)
https://reviews.freebsd.org/D54959
https://reviews.freebsd.org/D54960

Test Plan

Tested on a RISC-V IOMMU-enabled platform

[jrtc27]

How does this differ from the default implementation specified in sys/kern/bus_if.m, namely bus_generic_get_dma_tag?

[br]

In D55220#1262735, @jrtc27 wrote:

How does this differ from the default implementation specified in sys/kern/bus_if.m, namely bus_generic_get_dma_tag?

In the default implementation, the child device (for example, a virtio block drive) is passed up the device hierarchy to the PCI bus. This implementation, however, requests a tag at the Virtio-PCI level, treating the virtio pci controller (which is a device discovered on the PCI bus) as the effective child.

[jrtc27]

Oh I see, the point is about what's passed up as the second argument, not what device the method is invoked on. I guess then my question becomes why is the DMA tag special? We don't do this for resources, and we don't do this for the bus space tag. Is that just because normally it's virtio_pci doing the resource allocation?

[br]

In D55220#1262908, @jrtc27 wrote:

Oh I see, the point is about what's passed up as the second argument, not what device the method is invoked on. I guess then my question becomes why is the DMA tag special? We don't do this for resources, and we don't do this for the bus space tag. Is that just because normally it's virtio_pci doing the resource allocation?

Yes, second argument is Virtio PCI bus (which is a child of the PCI bus), so that PCI framework could find a context ID for address translation based on iommu-map property in DTS. Each PCI device has its own unique context ID.
We do that for DMA tag because that is how IOMMU works in freebsd: a default busdma backend provided by architecture is replaced with a generic IOMMU busdma backend. The IOMMU busdma backend manages the guest address space per domain and allows to establish and tear down arbitrary mappings by invoking methods on an underlying IOMMU driver (ARM System MMU, RISC-V IOMMU, x86 DMAR).

[jhb]

I would perhaps tweak the commit title to be something like "virtio pci: Enable device models on platforms IOMMUs" And then maybe in the first paragraph after the title use:

Request the bus_dma tag for the parent PCI bus device so that IOMMU drivers can determine the PCI RID of the device.

Note that another approach that might get to what Jessica is asking is that you could perhaps change these virtio_pci bus drivers to implement the pci_get_id method and have that just return the id from the parent device. This might fix more than just DMA operations, but other IOMMU things like interrupt remapping on x86 which uses pci_get_id directly. That is probably a better solution in fact since it is more general.

@manu @emaste: the DRM drivers should implement a similar wrapper pci_get_id method that returns the ID of the parent device which would remove the need for the gross hack in LinuxKPI currently around pci_get_id().

sys/dev/virtio/pci/virtio_pci_modern.c

196–198

[emaste]

In D55220#1269806, @jhb wrote:

@manu @emaste: the DRM drivers should implement a similar wrapper pci_get_id method that returns the ID of the parent device which would remove the need for the gross hack in LinuxKPI currently around pci_get_id().

cc @bz @dumbbell for that as well