pci: Add tunable to reserve bridge bus numbers

[kbowling]

Summary

Some firmware assigns only one bus number to each PCI-PCI bridge.
This prevents later SR-IOV VF enumeration when a VF routing ID falls
on another bus already allocated to a sibling bridge.

Add hw.pci.reserve_buses, a boot-time tunable for the desired minimum
bus-number count per bridge. The default of one preserves existing
behavior. Allocation remains best effort and can fall back to one bus.

MFC after: 1 month

Test Plan

Use case, some ewaste Pro Desk SFF machine to test NICs, has VT-d but no SR-IOV option firmware:

• Firmware assigned the 82576 root port only bus 1.
• The non-ARI 82576’s First VF RID Offset = 0x180 places its VFs on bus 2.
• Bus 2 was already assigned to the NVMe root port, so SR-IOV failed with ENOSPC.
• hw.pci.clear_buses=1 plus hw.pci.reserve_buses=2 produced non-overlapping windows 1–2 and 3–4.
• The NVMe remained operational and the extensive 82576 PF/VF matrix passed.

I have no idea if this extrapolates to other devices yet but it's also not intended for serious use.

[kib]

If this is a solution, then IMO it should be activated by default. How would user know to tweak this knob if SR-IOV fails?

What is the concern for not reserving some number of bus numbers behind each bridge? The possibility of exhausting the bus number space?

Also, suppose that we have bridges hierarchy where there is a switch behind switch. How would it work?

[mav]

I haven't worked with SR-IOV myself, so I've never seen this problem to seriously comment on it. I've tried to do re-allocation for other resources, but it always created some problems with devices already running at the time, but with resources not always even visible to the device tree, like console. I guess bus renumbering might be easier than memory, but for example, what happen with dmidecode output after that? Will data there no longer reflect the actual bus allocation? I saw some Supermicro motherboard BIOS'es, allowing to set resource reservation for all hot-plug-capable PCIe ports. Unfortunately those are quite rare, and again, cover only hot-pluggable ports. I am not too familiar with SR-IOV, but at least from the point of limited bus number space I wonder if this could be limited to buses with SR-IOV-capable devices, or something like that? Though it would still be a renumbering, unless you allocate the new bus ranges from a range above what BIOS already allocated, to minimize effects on unrelated devices. But if as Konstantin mentioned you have nested bridges, I suppose that would renumber the parents and siblings, which would be invasive and much harder.

[kbowling]

@jhb halp us