bhyve: avoid allocating BARs above the end of supported physical addresses

[kib]

Summary

Read CPUID leaf 0x8000008 to determine max supported phys address and create BAR region right below it, according to the total 64bit BARs limit.

Also temporary bump qualification size for BAR allocation in lower (< 4G) IO window to be 128M (it will be 256M in final separate commit).

PR: 250802

[grehan]

btw many thanks for doing this.

usr.sbin/bhyve/pci_emul.c

626 ↗

(On Diff #79190)

Comment should be 128

1114 ↗

(On Diff #79190)

This should be dropped down to the next lowest and largest power-of-2 address, since BAR allocations are always on pow2 boundaries.

[grehan]

usr.sbin/bhyve/pci_emul.c

1114 ↗

(On Diff #79190)

Maybe an easier way would be to start at the next 1G boundary past the end of configured guest RAM. and use from there until the end of phys address space for the region. That should cover just about any alignment/size.

[kib]

Change to 128 in comment.
Set mem64bar start address to max power of 2 less than calculated address.

[kib]

Dynamically calculate space we reserve for 64bit bars, set it to 1/4 of the max phys.
Previous model where fixed space was hardcoded did not worked because the value was too big for some CPUs.

[kib]

Fix formatting.

[kib]

usr.sbin/bhyve/pci_emul.c

1114 ↗

(On Diff #79190)

I experimented some with the suggestion, and I prefer the approach from the patch. I still need to calculate maxphysaddr to see how much memresv64 can be. And with the suggestion, I need to find the TOM as well.

[grehan]

This should be fine.

There is an additional issue that I can do in a follow-up commit, and that's the base address of 0x000000D000000000 which hard-coded in existing bhyve/UEFI ACPI DSDT. A simple fix is to use that address if it fits within existing address lines, and the proper fix will be to use the f/w i/f to extract the base addr/limit at runtime and patch the DSDT.

usr.sbin/bhyve/pci_emul.c

628 ↗

(On Diff #79234)

This bit of code can be removed: it was a relic leftover from supporting a Netapp 8G BAR NVRAM adapter that required a 1:1 GPA:HPA mapping for peer-peer DMA.

[kib]

usr.sbin/bhyve/pci_emul.c

628 ↗

(On Diff #79234)

I can do it as followup.

[kib]

Correct limit for membar64. It is abs address, not the size of the space to allocate from.

[kib]

Clamp cpu maxaddr at VM_MAXUSER_ADDRESS (LA47 variant due to only 4-level pages supported).
Also fix vmspace creation to use correct TOM for guest.

[kib]

Add headers. Fix typo LA47->LA48.

[grehan]

usr.sbin/bhyve/pci_emul.c

1118 ↗

(On Diff #79312)

It could be even simpler to just set pci_emul_membase64 to 3/4 * cpu_maxphysaddr

I should have been a bit clearer: the boundary doesn't have to be a power-of-2; just a multiple of a power-of-2, and that becomes the largest BAR size that could be placed. However, 3/4 of max phys addr is still an enormous number.

However, I'm fine with what you have.

[kib]

usr.sbin/bhyve/pci_emul.c

1118 ↗

(On Diff #79312)

I am working on the proper resource tracking for BAR allocation, in particular I already ported subr_vmem.c to userspace library. Having starting address be power of two is useful I believe, as the start of the resource region.