Use PCI power-mgmt to reset a device if FLR fails.

[grehan]

Summary

A large number of devices don't support PCIe FLR, in particular
graphics adapters. Use PCI power management to perform the
reset if FLR fails or isn't available, by cycling the device
through the D3 state.

This has been tested by a number of users with Nvidia and AMD GPUs.

[imp]

I click 'ok' with some reservations, however. Really old PCI devices may have issues with this, and the reset by going into D3 state isn't complete on some devices. However, despite those reservations, I think for the set of hardware that's relevant for FreeBSD and given that D3->D0 reset is the best we have for some devices it's a reasonable tradeoff. Since it's not in the bus code itself, it should present no problems.

[grehan]

bhyve passthru is currently restricted to the subset of devices that can support MSI/MSI-x so this doesn't come into play for older PCI devices.

[rgrimes]

Is this the workaround developed with macy?

[grehan]

Yes, this code was submitted by Matt M.

[jhb]

Yes, this is effectively the patch from Matt which I think I've reviewed before. I think this is fine. Some devices also support FLR via non-PCI-e means (PCI advanced features capability) but I don't think the GPUs do.

[imp]

In D15268#321853, @grehan wrote:

bhyve passthru is currently restricted to the subset of devices that can support MSI/MSI-x so this doesn't come into play for older PCI devices.

So much the better. All the rogues I know in this area are pre-PCIe, pre-MSI.