pci: Add hw.pci.intx_reroute sysctl/tunable

[cperciva]

Summary

INTRng can leak resources when INTx interrupts are re-routed, which is
typically harmless but can be fatal when devices are (repeatedly) hot
plugged into PCI buses on INTRng systems. Re-routing INTx interrupts
is nonetheless still necessary on some systems, and identifying whether
the re-routing should be enabled or disabled seems to be nontrivial.

Add a hw.pci.intx_reroute sysctl/tunable so systems which don't want
legacy PCI interrupt re-routing can turn it off. This is probably not
the best fix but it's something which can be safely included in FreeBSD
14.3.

[cperciva]

This is an alternative to https://reviews.freebsd.org/D49792

[cperciva]

@jhb ping?

[bz]

Silly question? In what way does this have anything to do with the kernel option I once set for arm64 for gem5? Is your problem specific to arm64? If it is the question is whether that option was needed by anything but gem5 ever. If it was not we could take it off again. I am fine with a tunable. I'd just love the default not being a double-negation. 1 is on, 0 is off, the feature is called "reroute interrupts".

sys/dev/pci/pci.c

432

Can you invert this? no_foo =flase as default is awkward. legacy_reroute = true

433

RDTUN or can we really change it at runtime?

[cperciva]

In D49849#1138905, @bz wrote:

Silly question? In what way does this have anything to do with the kernel option I once set for arm64 for gem5?

You set PCI_REROUTE_INTERRUPT on arm64, and then that option was later removed because it was being set on all supported architectures. Pre-PCI_REROUTE_INTERRUPT, we weren't calling pci_assign_interrupt(bus, dev, 1);.

Is your problem specific to arm64?

I'm tripping over it in EC2 arm64 instances. But it might exist in other places; I suspect that not many people have tested "do we leak IRQs if we hotplug/unplug devices dozens of times" on real hardware.

If it is the question is whether that option was needed by anything but gem5 ever. If it was not we could take it off again. I am fine with a tunable. I'd just love the default not being a double-negation. 1 is on, 0 is off, the feature is called "reroute interrupts".

Yeah I could change the sense here.

[cperciva]

sys/dev/pci/pci.c

433

We can definitely change it at runtime. I was doing exactly that while testing that it worked, in fact.

It only applies to devices which attach after the value changes, of course, but the place this matters to me is for (repeated) hotplug so that's just fine.

[bz]

Thanks for the history update; I was trying to figure it out and failed without starting to dig.
If @jhb is fine I would definitively be (if the double-neg goes away).

[jhb]

A better way to state the problem case might be that INTRng leaks resources when INTx interrupts are re-routed which can be fatal when PCI hot plug is used with INTRng.

sys/dev/pci/pci.c

433

Maybe rename this to pci_intx_reroute and default it to true, so:

static bool pci_intx_reroute = true
SYSCTL_BOOL(_hw_pci, OID_AUTO, intx_reroute, CTLFLAG_RWTUN,
    &pci_intx_reroute, 0, "Re-route INTx interrupts when scanning devices");

4182–4183

Paired with my other suggestion instead of below.

[cperciva]

no_legacy_reroute -> intx_reroute