Support for MSI-X on Annapurna Alpine

[mst_semihalf.com]

Summary

This patch adds support for MSI-X interrupts on Annapurna Alpine platform. MSI-X on Alpine work similarly to GICv2m, i.e. some range of SPI interrupts is reserved in GIC and individual SPIs can be triggered by MSI-X messages. This SPI range is defined in FDT.

I left in support for non-INTRNG kernel as INTRNG was not enabled on ARM64 at the time of writing this driver. However, I can remove it if INTRNG is to stay.
EDIT: I removed non-INTRNG parts of the driver.

[andrew]

However, I can remove it if INTRNG is to stay.

There is no way to disable INTRNG and get a working arm64 kernel.

[mst_semihalf.com]

In D7579#157268, @andrew wrote:

However, I can remove it if INTRNG is to stay.

There is no way to disable INTRNG and get a working arm64 kernel.

This driver is also used on ARM32 (there's a 32-bit Alpine SoC with the same devices) but INTRNG is enabled there as well so I'll remove non-INTRNG parts to make the patch easier to review.

[mst_semihalf.com]

Remove non-INTRNG parts of the driver as they are no longer necessary.

[mst_semihalf.com]

Use device entry instead of SOC option.

[nwhitehorn]

Is there any reason these bindings diverge from the ones from the vendor? http://lxr.free-electrons.com/source/arch/arm/boot/dts/alpine.dtsi

[mst_semihalf.com]

Rework MSI-X allocation to suit post-D7493 INTRNG.
I also removed some oddities leftover from old versions of the driver and simplified the code.

I added a small change to GICv3 (D7662) as BUS_SETUP_INTR was failing in the case of MSIX interrupts which were not extended with FDT description.

[mst_semihalf.com]

Move MSIX file entry from sys/conf/files to files.arm and files.arm64. Add fdt dependency.

[wma]

sys/arm/annapurna/alpine/alpine_pci_msix.c

160 ↗

(On Diff #19947)

Shouldn't this be an error? When we fail with registering device, we end up with non-working MSI/MSIx for the whole system and the driver successfully loaded.

[mst_semihalf.com]

Return error when MSI could not be registered.

sys/arm/annapurna/alpine/alpine_pci_msix.c

160 ↗

(On Diff #19947)

Fixed

[andrew]

sys/arm/annapurna/alpine/alpine_pci_msix.c

206–208 ↗

(On Diff #20065)

You seem to be requesting a contiguous range of interrupts and mapping them 1:1. This is the sort of thing intr_pic_add_handler was designed to handle. You might need to tell the GICv3 driver to send this range to the child.

313–320 ↗

(On Diff #20065)

Why not use vmem(9) to manage this? I've done so in the GICv3 ITS driver if you would like a reference.

325–332 ↗

(On Diff #20065)

Can you not use ofw_bus_map_intr?

[mst_semihalf.com]

sys/arm/annapurna/alpine/alpine_pci_msix.c

206–208 ↗

(On Diff #20065)

I know, but currently the child handler is called only in the case of LPIs so it only works with ITS on ARM64. I did not want to introduce delay in arm_gic_v3_intr() by adding additional branches, but perhaps it should somehow be resolved so that not only ITS is supported as child. Another potential issue is that the 'msix' node here is not really a child of GICv3 node in DTS - it is at the same level under 'soc'.

325–332 ↗

(On Diff #20065)

That only decodes the IRQ number, while PIC_MAP_INTR provides the isrc from GIC which we can return to the caller.

[mst_semihalf.com]

Change MSI-X number allocation scheme to make use of vmem_alloc() instead of local bitmap.

[mst_semihalf.com]

sys/arm/annapurna/alpine/alpine_pci_msix.c

313–320 ↗

(On Diff #20065)

Diff 21508 switches management to vmem. Had some trouble with it because vmem_alloc() seems to eventually fail when smallest unit of allocation is set to 1 and qcache_max is also 1. In ITS you always allocated more than one element at a time and I suppose that's why it worked for you - I think you should look into what happens when you try to allocate only one item. In my case a few initial allocations were successful but then it returned ENOMEM even though there was still a lot of free elements.

[wma]

Seems fixed