Page MenuHomeFreeBSD

vmm: keep memory decode enabled through MSI-X teardown
Needs ReviewPublic

Authored by nick_spun.io on Sun, Oct 11, 1:24 AM.
Tags
Referenced Files
F175869330: D60605.id189444.diff
Wed, Oct 14, 1:27 AM
F175852522: D60605.id189443.diff
Tue, Oct 13, 9:49 PM
F175852500: D60605.id189443.diff
Tue, Oct 13, 9:48 PM
F175848850: D60605.id189444.diff
Tue, Oct 13, 9:02 PM
F175848787: D60605.id189443.diff
Tue, Oct 13, 9:02 PM
F175848721: D60605.diff
Tue, Oct 13, 9:02 PM
Unknown Object (File)
Tue, Oct 13, 3:31 AM
Unknown Object (File)
Mon, Oct 12, 8:00 PM

Details

Reviewers
adrian
kbowling
markj
Group Reviewers
bhyve
Summary

ppt_unassign_device() and ppt_reset_device() cleared Memory Space
Enable before tearing down MSI-X. Masking a vector writes the MSI-X
table, which lives in a device BAR, so once decode is disabled the
write is an Unsupported Request. On a live guest (msix_alloc > 0)
the UR is reported through APEI as a fatal hardware error and the
host panics.

Tear down MSI/MSI-X with decode still on, then gate decode and DMA
before the reset.

See PCI Express Base Specification Revision 5.0 section 7.5.1.1.3
(Command register, Memory Space Enable)

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Tests Skipped
Build Status
Buildable 77986
Build 74869: arc lint + arc unit

Event Timeline

Makes sense: ppt_teardown_msix_intr() -> bus_teardown_intr() -> pci_mask_msix() writes the vector control dword in the table BAR, so with decode off that is a UR. One suggestion: the old ordering also stopped DMA before the interrupt teardown (the device could still be bus-mastering while its guest mappings go away), and the new one leaves BUSMASTEREN on until afterwards. Clearing BUSMASTEREN together with INTxDIS first, and only deferring PORTEN/MEMEN until after the MSI-X teardown, keeps both properties. (MSI teardown doesn't touch BARs; only MSI-X does.) We carry the same ordering in keelOS's ppt (stable/15) and will pick this up.