Reset PCI pass through devices via PCI-e FLR during VM start and end.

[jhb]

Summary

Reset PCI pass through devices via PCI-e FLR during VM start and end.

Add routines to trigger a function level reset (FLR) of a PCI-express
device via the PCI-express device control register. This also includes
support routines to wait for pending transactions to complete as well
as calculating the maximum completion timeout permitted by a device.

Change the ppt(4) driver to reset pass through devices before attaching
to a VM during startup and before detaching from a VM during shutdown.

Test Plan

• Passed a Chelsio T5 VF through to a bhyve VM. The VF works ok inside the VM and works again if I start a new one. Added some log messages to verify that it did do the writes to initiate a FLR, but I have not been able to confirm via the devlog of the parent PF if the FLR was noticed. (It's not clear if the firmware on the T5 logs messages for PCI-e initiated FLR requests. It does for device resets requested via a device firmware request.)

[rpokala]

sys/dev/pci/pci.c

5955 ↗

(On Diff #19958)

return (50 * 1000); /* perhaps? */

[wblock]

share/man/man9/pci.9

450 ↗

(On Diff #19958)

s/If/When/ to avoid repetition with the previous sentence.

478 ↗

(On Diff #19958)

If/then sentences can be tedious for the reader. Some can be rearranged to avoid the built-in pause, like this one:

.Nm pcie_wait_for_pending_transactions
returns
.Dv true
if
.Fa dev
is not a PCI-express device.

This is the equivalent of a Perl/Ruby postfix if, really. Or rather, those are the equivalent of this type of wording.

722 ↗

(On Diff #19958)

Some of the redundancy can be reduced, here and elsewhere:

.Fn pcie_flr
requests a Function Level Reset
​.Pq FLR
​of
.Fa dev .

727 ↗

(On Diff #19958)

Maybe just

.Dv false
is returned.``

(removing "this function")

731 ↗

(On Diff #19958)

Yes! This is better.

743 ↗

(On Diff #19958)

Maybe just

busmastering is re-enabled and
.Dv false
is returned.

[jhb]

• Fix value of default timeout.
• Various wording suggestions from Warren.

[imp]

Looks good to me (though I defer to Warren on the docs).

[jhb]

• Expand pause_ms() and remove it.

[jhb]

Rebase.