Make VF creation more robust.

[jhb]

Summary

Make VF creation more robust.

• Use the correct location of the SRIOV capability when enabling ARI. Also, if enabling ARI fails, fail to create VFs.
• Save the value of the IOV control and page size registers and restore them (along with the VF count) in pci_cfg_save/pci_cfg_restore. This ensures ARI remains enabled if a PF driver resets itself during the PCI_IOV_INIT callback. Theoretically this might also properly restore SRIOV state across suspend/resume.

(Note: I can split this into two commits, but might be easier to review
in one sitting. I also tested them together.)

Test Plan

• Try to create VFs on an out-of-tree driver that resets the adapter in the PCI_IOV_INIT callback resulting in the IOV_CTL register being cleared and the ARI setting being lost. This means config access to the VFs doesn't work after they are created.

[rstone]

sys/dev/pci/pci_iov.c

455 ↗

(On Diff #15865)

Oof. Good catch.