pcib: Allocate the memory BAR with the MSI-X table.

[jhb]

Summary

This is required for pci_alloc_msix() to work and to thus use
MSI-X interrupts for PCI-e hotplug.

Reported by: cperciva

[cperciva]

FWIW, this makes it possible for an ena device attached to a PCIe hotplug port to attach, but it's getting watchdog timeouts so there's something else going on which might or might not be related to PCIe/MSIX.

[jhb]

I'm pretty sure that Colin's unique device is the first PCI-e HP bridge that supports MSI-X but not MSI. We have probably just always been falling back to MSI before without anyone noticing.

[cperciva]

In D40581#923957, @cperciva wrote:

FWIW, this makes it possible for an ena device attached to a PCIe hotplug port to attach, but it's getting watchdog timeouts so there's something else going on which might or might not be related to PCIe/MSIX.

It sounds like the "something else going on" is a hypervisor bug which is being fixed separately. So as far as FreeBSD is concerned the patch we have here should be the only thing we need.

Is this MFCable? It breaks ABI but it's an ABI in pcib_private.h which maybe we don't care about that?

[jhb]

It is definitely MFCable, the pcib softc is not a public KBI.