When this was applied on top of illumos, I observered the follwing

• VNC works fine without a pci device in passthru
• VNC was was corrupted and OS failed to boot when passtrhu of a PCIe device with multi BARs (a X722 Nic) https://camo.githubusercontent.com/e5b5ba02421afe404fdb60ad68ef3cfe0e86cc90/68747470733a2f2f696d6775722e636f6d2f6d63694a724b4d2e706e67

I'm assuming this is an illumos specific issue, but it might be worth to test this on FreeBSD too.

I'm assuming this is an illumos specific issue, but it might be worth to test this on FreeBSD too.

It will happen on FreeBSD as well. passthru devices don't allow remapping, and that's why the original UEFI firmware had enumeration disabled.

It could well be worth investigating if the changes to UEFI to properly support PcdPciDisableBusEnumeration = TRUE are less invasive than allowing runtime BAR changes
(Intel ACRN has this as TRUE)

In D24066#529470, @grehan wrote:

It will happen on FreeBSD as well. passthru devices don't allow remapping, and that's why the original UEFI firmware had enumeration disabled.

Interesting, without the change form https://github.com/freebsd/uefi-edk2/pull/9/ these NICs are working fine with passthru. (Not tried NVMe as this is not yet supported on illumos).

So it will be a choice of a UEFI firmware that does VNC and passthru but no NVMe or one that does VNC and NVMe, that will be confusing.

So it will be a choice of a UEFI firmware that does VNC and passthru but no NVMe or one
that does VNC and NVMe, that will be confusing.

The older UDK2014.1-based firmware works fine with all combinations; the regressions in the sync just need to be fixed.

In D24066#529470, @grehan wrote:

It could well be worth investigating if the changes to UEFI to properly support PcdPciDisableBusEnumeration = TRUE are less invasive than allowing runtime BAR changes
(Intel ACRN has this as TRUE)

I haven't looked into the edk2 code to see what relaxing its expectations would look like, but one much simpler way to work around the heartburn it gets from the nvme model is to just change its bar from MEM_64 to MEM_32: https://gist.github.com/d-scott-phillips/d71d8d0c856301e5738ee73b68e56ad4 (I'm not sure if the nvme spec requires the bar to be 64-bit capable.)

(I'm not sure if the nvme spec requires the bar to be 64-bit capable.)

Looks like it's only recommended ("2.1.10 Offset 10h: MLBAR (BAR0) – Memory Register Base Address, lower 32-bits") so forcing 32-bit could be a quick fix.

In D24066#533961, @scottph wrote:

I haven't looked into the edk2 code to see what relaxing its expectations would look like, but one much simpler way to work around the heartburn it gets from the nvme model is to just change its bar from MEM_64 to MEM_32: https://gist.github.com/d-scott-phillips/d71d8d0c856301e5738ee73b68e56ad4 (I'm not sure if the nvme spec requires the bar to be 64-bit capable.)

I've not been following this review, but I did run into a problem with bhyve's use of the MEM64 BAR recently with my work on updating UEFI firmware to edk2-stable202002. It turns out edk2 used to assume that MEM64 BARs were located above 4GB. That problem has since been fixed, and I've found NVMe to work well now. I've not however tested PCI passthrough yet.

That problem has since been fixed, and I've found NVMe to work well now.

Is this with PcdPciDisableBusEnumeration = TRUE in UEFI ?

In D24066#534662, @grehan wrote:

Is this with PcdPciDisableBusEnumeration = TRUE in UEFI ?

Yes.

Is this waiting on anything?

In D24066#616957, @emaste wrote:

Is this waiting on anything?

I've been meaning to spend some spare time on re-testing various
cases to make sure it hasn't broken something in the meantime. If
somebody has time to test out fbuf and passthru cases both with
and without https://github.com/freebsd/uefi-edk2/pull/9/ in the
uefi, that could help speed things along.

The patch no longer applies cleanly: I had to make some changes.
But after that, I was able to run Ubuntu 20.10 with gEfiMdeModulePkgTokenSpaceGuid.PcdPciDisableBusEnumeration set to TRUE and FALSE with the following command line, which uses both rfb and passthru:

bhyve -AH -s 0:0,hostbridge -s 31:0,lpc -c 1 -m 16G -l bootrom,BHYVE_CODE.fd -l com1,stdio -s 3:0,ahci-hd,bhyve-ubuntu-20_10.img  -s 7,ahci-cd, -s 8,virtio-rnd -s 6,virtio-net,tap0 -s 9,fbuf,rfb=10.0.10.117:5900,w=1920,h=1200 -s 10,xhci,tablet -s 11,passthru,24/0/0 -S guest

Sorry for the late reply. I am running through one of the reports which triggers the assert at the end of this function in 13.0-R and wondered why there is no error != 0 checks for the changes in pci_emul.c