Set domain.end address (max guest addr + 1) to the VM_MAXUSER_ADDR

Fix comment

Rename iommu_smmu.c to smmu_iommu.c

The final patch to review

Address markj@ queries

Add default implementations of iommu_init/deinit

gicv3_iommu_deinit() added

o Check if entry is not NULL, otherwise return early
o Remove gicv3_its.c changes from this diff. They are moved to https://reviews.freebsd.org/D26878

Move iommu stubs to subr_bus_dma.c

In D26887#599735, @andrew wrote:

Would it make sense to move these to subr_bus_dma.c in the non-IOMMU case?

o Rename pmap_sremove_all() to pmap_sremove_pages()
o Remove unneeded check for L2_BLOCK in pmap_sremove_pages()

Add IOMMU quirks for all the devices listed in linux

Update AHCI quirk bitstring

Move the quirk to ahci driver

o Add a KASSERT
o Mark MSI fields in the iommu_domain as arch-specific

Regenerate

o Move iommu_get_ctx_domain(), iommu_get_dev_ctx() to iommu.h

In D26705#598296, @kib wrote:

I am curious. There is no arch restriction on the value of MSI base address ? System can map MSI page anywhere, and just programming the correct address into MSI address register would make it work ?

This is very non-x86ish.

o Put Andrew's copyright
o Remove ifdef IOMMU

We could also note that the amd64's IOMMU subsystem was splitted-out from amd64 DMAR support and is now generic, i.e. could be used by all architectures

Regenerate

Address kibs's notes

Address kib's notes:

• add iommu_types.h
• add md macro IOMMU_UNLOAD_SLEEP

context added

In D26705#595414, @br wrote:

In D26705#595261, @kib wrote:

So would adding a header iommu_msi.h with just msi-related prototypes enough ? I suspect that x86/msi.c can also get rid of iommu.h then.

I will try and see if it works

Manage MSI iommu pages

In D26705#595261, @kib wrote:

So would adding a header iommu_msi.h with just msi-related prototypes enough ? I suspect that x86/msi.c can also get rid of iommu.h then.

In D26705#595130, @kib wrote:

Convention for var.h is not suitable for your case IMO. We have signal.h and signalvar.h, where signal.h is usable by userspace and signalvar.h is for kernel private stuff.

Can you provide some enumeration of stuff you want to keep in iommu.h, and an explanation of why ?

a change by andrew@:

In D24618#583893, @mmel wrote:

I’m slightly confused by code partitioning.
With that in mind that we have multiple types of IOMMUs (rockchip, tegra, smmu v2, allwinner …, and one SoC can have multiple different IOMMUs, I’m not sure how this patch fit in this situation.
Is iommu.c intended as base for future IOMMU independent part of framework? If yes, why it contain functions prefixed by smmu_? If not, why it’s named by generic name?
Do you have any roadmap, ideas how final (supporting multiple kinds of IOMMUs in one generic kernel) should be structured?

Regenerate

Don't repeat iommu_get_dma_tag() declaration, include iommu.h instead.

Thanks kib@: buswide quirk works, I have updated the main diff:
https://reviews.freebsd.org/D24618

Regenerate

Add quirk: iommu set to buswide for Marvell AHCI

In D26360#589183, @kib wrote:

Yes, I was aware of some AHCI controllers mis-handling RIDs. It is the duty of the driver to ensure that proper domain is created which includes all relevant contexts (not that we have an KPI for this right now).

But wait, why cannot you use iommu_set_buswide_ctx() there ?

Also here are Linux quirks for Marvell:
https://elixir.bootlin.com/linux/latest/source/drivers/pci/quirks.c#L4026

In D26360#588838, @kib wrote:

In D26360#588657, @br wrote:

In D26360#588498, @kib wrote:

In D26360#587442, @br wrote:

In D26360#587123, @kib wrote:

Well, this is not the right approach. I am guilty of the buswide_ctx() of course, but this one goes much further.

Proper solution is to finish my patch to support DMAR (IOMMU ?) domains, where single domain with its page table serves some number of contexts. Then multi-function devices that must share translations should be put into common domain instead of creating per-function domains.

We don't create a per-function domain. This is just to setup a Stream Table Entry (STE) alias in the 2-level walk table.
Marvel SATA issues non-dma requests from SID 0x600, and DMA from 0x601. So SMMU simply could not find a valid STE in the position of 0x601 in the 2-level walk table.
The solution is just to copy the same STE to a required position in the walk table and everything works (this is what Linux does as well).

In essence, you just reworded what I wrote.

Let me be more explicit: I believe such cases should be handled by creating domains which contain more than one context.

Do you mean that each function alias should be in a separate iommu context?

There are no aliases in PCIe spec or DMAR. Yes, DMAR spec makes 1:1 correspondence between context and RID. Domain can group several contexts that share page tables, making all RIDs from domain using identical translations. I believe AMD IOMMU is same.

In D26360#588498, @kib wrote:

In D26360#587442, @br wrote:

In D26360#587123, @kib wrote:

Well, this is not the right approach. I am guilty of the buswide_ctx() of course, but this one goes much further.

Proper solution is to finish my patch to support DMAR (IOMMU ?) domains, where single domain with its page table serves some number of contexts. Then multi-function devices that must share translations should be put into common domain instead of creating per-function domains.

We don't create a per-function domain. This is just to setup a Stream Table Entry (STE) alias in the 2-level walk table.
Marvel SATA issues non-dma requests from SID 0x600, and DMA from 0x601. So SMMU simply could not find a valid STE in the position of 0x601 in the 2-level walk table.
The solution is just to copy the same STE to a required position in the walk table and everything works (this is what Linux does as well).

In essence, you just reworded what I wrote.

Let me be more explicit: I believe such cases should be handled by creating domains which contain more than one context.

In D26360#587123, @kib wrote:

Well, this is not the right approach. I am guilty of the buswide_ctx() of course, but this one goes much further.

Proper solution is to finish my patch to support DMAR (IOMMU ?) domains, where single domain with its page table serves some number of contexts. Then multi-function devices that must share translations should be put into common domain instead of creating per-function domains.

In D26360#586327, @andrew wrote:

Why is it in the pci code rather then iommu? I could imagine broken non-pci devices could also need a DMA alias.

Regenerate

Check the return value of smmu_map_msi()