Looks good

Reuse DMAR busdma backend for SMMU with minimal changes. This backend completely eliminates bouncing. Changes to DMAR are:
o Rename dmar to iommu
o Rename ctx to iommu_device
o Some dmar-specific functions removed

we should also need to add these devices to riscv kernel config (at least random and console)

I tested in tinyemu. It works.
(tinyemu supports version 2 only)

Correct the size of asid bit set.

o Since we don't share TLB with the main CPU, implement our own ASID allocator.
o Add smmu_tlbi_asid() function that allows to invalidate entire SMMU TLB for a specific ASID.

In D24618#542293, @kib wrote:

I have very generic questions before I ever start reading that.

1. Why did you put the stuff into sys/dev/iommu while it is arch-specific ? I decided to go with sys/x86/iommu.

I think keeping all IOMMU engines and the framework in a single place is a good idea. It is easy to find all the implementations. Similar to dev/mmc/host, dev/usb/controller etc. But I'm happy with arm64/iommu too.

Add context.

Add a hardware overview to the header of smmu.c

Set ASID to a Context Descriptor. The ASID is generated by pmap.c and stored to pmap->pm_cookie.
This is a quick fix for the issue I had with TLB.
We need to check if SMMU asid space is equal to pmap asid space (they both either 8 or 16 bits wide).

In D24499#538942, @mhorne wrote:

In D24499#538813, @br wrote:

What is the reason for a fatal page fault when some CPUs are not fired?
Normally it should start with a single CPU in that case.359280

I observed the page fault when the scheduler tried to run a thread on an absent CPU. I wrote this patch a couple weeks ago, and @markj changed the behaviour since then in rS359280. Now, if we time-out the system will hang in smp_after_idle_runnable.

What is the reason for a fatal page fault when some CPUs are not fired?
Normally it should start with a single CPU in that case.

Fix spelling

Compare to NULL

Add a call to panic()

In D24018#528102, @jhb wrote:

Hmm, I'm curious if this breaks on other platforms then.

Add context

In D23877#527473, @philip wrote:

It sounds like we're reaching consensus that we should move forward with this.

I'll do another build test before committing this patch-set later today. Unless someone screams. :)

Hi Nicholas. I'm doing efforts to figure out why this is needed.
Why cpu_check_mmu() does not work for you and what are specific devmap use cases you have at Axiado?

The pcie_discovery_data is device specific, move it to the softc.
Move the root memory to the softc and switch it to use bus_space.
While here create a common function to get the correct bus_space
details for a given bus/slot/func/reg combination.

Move the driver to dev/pci/controller/

what about dev/pci/controller/? linux does that.
We have a similar subdirectory for MMC drivers: dev/mmc/host/ that works well and keeps dev/mmc/ clean

move the driver by request from jhb@

In D23349#516026, @jhb wrote:

As with the xilinx driver, can we please not put MD and chipset-specific code in sys/dev/pci?

o Address mmel's comment.
o Don't override MSI methods since Andrew has fixed a bug in the ITS driver

Include pcI_n1sdp to the build

Add comment for MSI/MSIx

Override MSI/MSIx pcib methods so MSI allocation fail and INTx are used.
MSI/MSIx don't work for now

Address Andrew's comments: use kva_alloc/pmap_enter for non-device memory

Looks good

nice catch!

Rename the driver to dev/xilinx/xlnx_pcib.[ch] per request from jhb@

Looks good. Thank you!

Looks good.

Looks good

I also have a comment from jhb@:
@br typically we only route device interrupts to a single core since interrupt handlers generally assume they are not run concurrently (interrupt filters, ithreads will be single-threaded even if the interrupt is broadcast)
as Andy noted, you want to handle interrupt binding requests, but you also need to do something for the “default” case.
Not sure what INTRNG does, but on x86 we round-robin interrupts among CPUs when they are first setup.

In D21928#484096, @mhorne wrote:

In D21928#484089, @br wrote:

Why do we need to enable interrupts for all CPUs and not only current cpu?
where an interrupt will be signaled in that case?

PIC_ENABLE_INTR is called very seldomly, often only once while setting up the PIC during intr_setup_irq by the BSP. Since it is not called per-cpu, this function should handle enabling the interrupt for all active CPUs.

BBL sets all PLIC enable bits to 1 by default, whereas OpenSBI sets them to 0, which is how I noticed that APs weren't having their interrupts enabled.

looks good!

Why do we need to enable interrupts for all CPUs and not only current cpu?
where an interrupt will be signaled in that case?

Are these example of PLIC contexts?

interrupts-extended = <
				&cpu0_intc 0xffffffff
				&cpu1_intc 0xffffffff &cpu1_intc 9
				&cpu2_intc 0xffffffff &cpu2_intc 9
				&cpu3_intc 0xffffffff &cpu3_intc 9
				&cpu4_intc 0xffffffff &cpu4_intc 9>;