Use IOAPIC PCI rid as the interrupt TLP source id for DMAR interrupt remapping.
ClosedPublic

Authored by kib on Sep 1 2017, 1:56 PM.

Details

Summary

If IOAPIC has PCI binding, match the PCI device against MADT enumerated IOAPIC. Match is done first by registers window phys address, then by IOAPIC ID.

When calculating rid for remapping interrupts sourced from the IOAPIC, first try to get the PCI rid, otherwise use rid as specified in the DMAR scope entry (i.e. the only current way to get rid for IOAPIC).

Test Plan

Tested this on desktop-class machine where IOAPIC does not make PCI appearence, and on the Skylake server. There currently the DMAR scope entries for uncore IOAPICs still have length two and since no bus renumbering occured, the PCI rid and scope rid are equal.

I checked that legacy interrupts still work (MSI/MSI-X were disabled manually for xhci):

flix1a% vmstat -i | grep -v cpu     
interrupt                          total       rate
irq0: attimer0                      2434         21
irq3: uart1                          247          2
irq16: xhci0                         292          3
irq257: dmar0:qi                       1          0
irq259: dmar1:qi                       1          0
irq261: dmar2:qi                   79202        690
irq263: dmar3:qi                       1          0
irq265: dmar4:qi                       1          0
irq267: dmar5:qi                       1          0
irq269: dmar6:qi                       1          0
irq271: dmar7:qi                     141          1
irq280: em0:rxq0                   70915        618
irq281: em0:rxq1                     218          2
irq282: em0:aq                         2          0
Total                             290485       2530

Also apparently PCI BARs for IOAPICs are not mapped. The attach process triggers the resource allocation.

flix1a% dmesg | grep BAR0
ioapic0: pci0:133:5:4 pci BAR0@e0f00000 id 0 MADT id 0 paddr@fec00000
ioapic1: pci0:174:5:4 pci BAR0@ee700000 id 1 MADT id 1 paddr@fec01000
ioapic2: pci0:215:5:4 pci BAR0@fbf00000 id 2 MADT id 2 paddr@fec08000

Diff Detail

Repository
rS FreeBSD src repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
kib created this revision.Sep 1 2017, 1:56 PM
kib edited the test plan for this revision. (Show Details)Sep 1 2017, 2:06 PM
kib edited the test plan for this revision. (Show Details)
kib added a reviewer: jhb.
kib added a subscriber: emaste.
emaste added inline comments.Sep 1 2017, 2:38 PM
sys/x86/iommu/intel_drv.c
826 ↗(On Diff #32585)

If this is to eliminate the unused variable warning I think (void)error; is more conventional, but why not just eliminate the variable?

kib marked an inline comment as done.Sep 1 2017, 2:55 PM
kib added inline comments.
sys/x86/iommu/intel_drv.c
826 ↗(On Diff #32585)

I did not wanted the #ifdef clutter in the declaration section. But ok, redid it.

kib updated this revision to Diff 32588.Sep 1 2017, 2:55 PM
kib marked an inline comment as done.

Hide the variable declaration under #ifdef.

jhb accepted this revision.Sep 8 2017, 6:34 PM
This revision is now accepted and ready to land.Sep 8 2017, 6:34 PM
This revision was automatically updated to reflect the committed changes.