bhyve: Manage passthrough devices across guest FLR

[kbowling]

Summary

bhyve emulates the guest PCI Command register so BAR sizing does not
disable physical decoding.  However, PCIe Device Control was passed
through.  A guest VFIO reset therefore performed a physical FLR, which
cleared physical Command, while the guest restored only its emulated
copy.  The device remained assigned with bus mastering disabled and
could not fetch DMA descriptors.

Intercept guest FLR writes and issue a PPT-managed reset.  Stop all
vCPUs, verify ownership, quiesce the function, perform only an FLR, and
restore the host-owned PCI configuration, decode, and bus-master state.
Keep the IOMMU domain in place.  bhyve removes guest BAR mappings before
this ioctl; a later guest MEMEN write recreates them.  Never escalate a
guest FLR to a power reset.

Reset the guest-owned Command, MSI, MSI-X, MSI-X table, INTx, and MRRS
state.  PCIe 6.2 section 6.6.2 explicitly preserves MPS across FLR.
Virtualize MPS, MRRS, and Completion Timeout.  Keep physical MPS and
completion-timeout policy host-owned, and apply physical MRRS with MPS
as its floor.  Keep Phantom Functions Enable host-owned because it
changes requester identities visible to the IOMMU.

Serialize guest configuration transactions per function and gate
trapped and direct BAR access across the reset.  Handle byte, word,
dword, and overlapping Device Control accesses.

A guest FLR can sleep for at least 100 ms.  Reserve the target function
while dropping the global PPT lock so a guest cannot delay PPT lifecycle
operations for other VMs.  Operations on the target wait for its reset
while other functions and VMs can proceed.

Check pcie_flr_supported() before destructive preparation so PPT applies
the generic PCI quirk policy.  This includes VFs such as the 82599 which
implement FLR without advertising it.

Validated with two E610 VFs in a Linux 7.0 guest using VFIO no-IOMMU and
DPDK testpmd with two queues per VF.  Byte, word, dword, and overlapping
FLR writes, 32 alternating resets, DPDK traffic, and ixgbevf
reattachment all completed while the sibling VF and host PCIe remained
healthy.  This fixes Linux VFIO no-IOMMU with DPDK PMDs.

MFC after:      2 weeks

Test Plan

DPDK testpmd on a Linux guest with VFIO no-IOMMU

[kbowling]

sys/amd64/vmm/io/ppt.c

503

This is specifically coded so the expense of the FLR only penalizes the requesting VM. A future headache might try to make this asynchronous to the VM's vCPUs but needs significant lifecycle auditing and this is not typically a common operation.

[markj]

sys/amd64/vmm/io/ppt.c

266

I think all of the callers could instead sleep until the FLR is finished. Why not do that everywhere, rather than introducing a new interface?

[markj]

sys/amd64/vmm/io/ppt.c

706

Why do you need this check? ppt_teardown_msi() and ppt_teardown_msix() will block until the associated handler is finished running, so I can't see what race this solves. This is the only access of ppt->resetting that isn't synchronized by the global PPT lock, so if you remove this, there's no need for the atomic_* qualified accessors either.

[kbowling]

sys/amd64/vmm/io/ppt.c

706

It attempts this window

1. resetting becomes true.
2. An interrupt arrives before MSI/MSI-X teardown completes.
3. pptintr() runs and drops it.
4. bus_teardown_intr() then drains the filter.

Without it, I think a stray MSI interrupt could survive reset at step 3 through a reset

[markj]

sys/amd64/vmm/io/ppt.c

706

The race is still there, this check just makes it less likely; consider that an NMI might arrive after pptintr() observed that ppt->resetting is false, and ppt_reset_device(). But why is it a problem to begin with?

[markj]

Looks ok to me with comments addressed.

usr.sbin/bhyve/pci_passthru.c

1271

I can't see a reason to save errno here. The caller ignores its value.

1651

I don't really understand the purpose of the unlocked checks. The vast majority of the time, psc_resetting will be false, so what's the point? And, if you remove them, then you don't need the atomics.

[kbowling]

address feedback

[kbowling]

Commited as c8343ba84af75c3710a1c587ff9bd2165fcd37a9