Add AMD IOMMU support in bhyve

[anish]

Summary

Enable basic pass-through or direct assignment of devices to guest on AMD with bhyve.

By default it is disabled unless enabled by setting hw.vmm.amdvi_enable=1

Continuation of https://reviews.freebsd.org/D9921

Test Plan

Tested on very Phenom with IOMMU h/w ver1.
Also tested by Rod on various h/w.

[grehan]

Tested fine on a Ryzen 1700 + Asus X370 mobo, with a dual-port Intel 82571 card. Both ports passed through to a FreeBSD 12-CURRENT amd64 4 vCPU guest, with iperf3/netperf stress tests run.

[grehan]

The most recent spec (Rev 3.00—December 2016, http://support.amd.com/TechDocs/48882_IOMMU.pdf) seems to indicate that v1 is the base features, and new features are described as options, so not sure how to determine if it's v2. The IVRS table does have a v1/v2 format discriminator, though that isn't printed out in the current code.

[bcr]

OK from the doc side.

[anish]

v2 extended feature is in "MMIO Offset 0030h IOMMU Extended Feature Register".

• If no extended feature[EFR], its rev1 with maximum paging level as 7. */ max_ptp_level = 7; if (softc->ivhd_efr) { device_printf(dev, "EFR HATS=%d GATS=%d GLXSup=%d" "MsiNumPr=%d PNBanks=%d PNCounters=%d\n" "PASID max =%d EFR:%b \n", REG_BITS(softc->ivhd_efr, 31, 30), REG_BITS(softc->ivhd_efr, 29, 28), REG_BITS(softc->ivhd_efr, 4, 3), REG_BITS(softc->ivhd_efr, 27, 23), REG_BITS(softc->ivhd_efr, 22, 17), REG_BITS(softc->ivhd_efr, 16, 13), REG_BITS(softc->ivhd_efr, 12, 8), softc->ivhd_efr, "\020\001XTSup\002NXSup\003GTSup\005IASup" "\006GASup\007HESup\008PPRSup");
  
  max_ptp_level = REG_BITS(softc->ivhd_efr, 31, 30) + 4; }

[grehan]

Thanks for the clarification on v2.