Page MenuHomeFreeBSD
Differential D27095

bhyve: avoid allocating BARs above the end of supported physical addresses
ClosedPublic
Actions

Authored by kib on Nov 5 2020, 12:49 AM.
Tags
None
Referenced Files
Unknown Object (File)
Sun, Dec 1, 10:21 PM
Unknown Object (File)
Nov 21 2024, 9:16 PM
Unknown Object (File)
Nov 14 2024, 2:46 PM
Unknown Object (File)
Nov 13 2024, 3:33 PM
Unknown Object (File)
Nov 2 2024, 2:30 AM
Unknown Object (File)
Oct 30 2024, 12:11 AM
Unknown Object (File)
Oct 29 2024, 5:41 PM
Unknown Object (File)
Oct 4 2024, 1:31 PM
View All Files
Subscribers
afedorov
bcran
crowston_protonmail.com
imp
rgrimes

Details

Reviewers
grehan
Group Reviewers
bhyve
Commits
rS367607: bhyve: increase allowed size for 64bit BAR allocation below 4G from 32 to 128…
rS367606: bhyve: avoid allocating BARs above the end of supported physical addresses.
Summary

Read CPUID leaf 0x8000008 to determine max supported phys address and create BAR region right below it, according to the total 64bit BARs limit.

Also temporary bump qualification size for BAR allocation in lower (< 4G) IO window to be 128M (it will be 256M in final separate commit).

PR: 250802

Diff Detail

Repository
rS FreeBSD src repository - subversion
Lint
Lint Skipped
Unit
Tests Skipped
Build Status
Buildable 34658

Event Timeline

kib created this revision.Nov 5 2020, 12:49 AM
Herald added a reviewer: bhyve. · View Herald TranscriptNov 5 2020, 12:49 AM
Herald added subscribers: bcran, rgrimes, imp. · View Herald Transcript
kib requested review of this revision.Nov 5 2020, 12:49 AM
Harbormaster completed remote builds in B34633: Diff 79190.Nov 5 2020, 12:50 AM
grehan added a comment.Nov 5 2020, 12:54 AM

btw many thanks for doing this.

usr.sbin/bhyve/pci_emul.c
625

Comment should be 128

1113

This should be dropped down to the next lowest and largest power-of-2 address, since BAR allocations are always on pow2 boundaries.

grehan added inline comments.Nov 5 2020, 1:03 AM
usr.sbin/bhyve/pci_emul.c
1113

Maybe an easier way would be to start at the next 1G boundary past the end of configured guest RAM. and use from there until the end of phys address space for the region. That should cover just about any alignment/size.

kib updated this revision to Diff 79192.Nov 5 2020, 1:16 AM

Change to 128 in comment.
Set mem64bar start address to max power of 2 less than calculated address.

Harbormaster completed remote builds in B34634: Diff 79192.Nov 5 2020, 1:16 AM
crowston_protonmail.com added a subscriber: crowston_protonmail.com.Nov 5 2020, 1:12 PM
afedorov added a subscriber: afedorov.Nov 5 2020, 1:53 PM
kib updated this revision to Diff 79233.Nov 5 2020, 9:42 PM

Dynamically calculate space we reserve for 64bit bars, set it to 1/4 of the max phys.
Previous model where fixed space was hardcoded did not worked because the value was too big for some CPUs.

Harbormaster completed remote builds in B34653: Diff 79233.Nov 5 2020, 9:42 PM
kib updated this revision to Diff 79234.Nov 5 2020, 9:44 PM

Fix formatting.

Harbormaster completed remote builds in B34654: Diff 79234.Nov 5 2020, 9:44 PM
kib marked 2 inline comments as done.Nov 5 2020, 9:45 PM
kib added inline comments.
usr.sbin/bhyve/pci_emul.c
1113

I experimented some with the suggestion, and I prefer the approach from the patch. I still need to calculate maxphysaddr to see how much memresv64 can be. And with the suggestion, I need to find the TOM as well.

grehan accepted this revision.Nov 5 2020, 9:54 PM

This should be fine.

There is an additional issue that I can do in a follow-up commit, and that's the base address of 0x000000D000000000 which hard-coded in existing bhyve/UEFI ACPI DSDT. A simple fix is to use that address if it fits within existing address lines, and the proper fix will be to use the f/w i/f to extract the base addr/limit at runtime and patch the DSDT.

usr.sbin/bhyve/pci_emul.c
628

This bit of code can be removed: it was a relic leftover from supporting a Netapp 8G BAR NVRAM adapter that required a 1:1 GPA:HPA mapping for peer-peer DMA.

This revision is now accepted and ready to land.Nov 5 2020, 9:54 PM
kib added inline comments.Nov 5 2020, 10:04 PM
usr.sbin/bhyve/pci_emul.c
628

I can do it as followup.

kib updated this revision to Diff 79243.Nov 6 2020, 12:01 AM

Correct limit for membar64. It is abs address, not the size of the space to allocate from.

This revision now requires review to proceed.Nov 6 2020, 12:01 AM
Harbormaster completed remote builds in B34658: Diff 79243.Nov 6 2020, 12:02 AM
kib updated this revision to Diff 79310.Nov 7 2020, 9:20 PM

Clamp cpu maxaddr at VM_MAXUSER_ADDRESS (LA47 variant due to only 4-level pages supported).
Also fix vmspace creation to use correct TOM for guest.

Harbormaster completed remote builds in B34689: Diff 79310.Nov 7 2020, 9:20 PM
kib updated this revision to Diff 79312.Nov 7 2020, 10:44 PM

Add headers. Fix typo LA47->LA48.

Harbormaster completed remote builds in B34690: Diff 79312.Nov 7 2020, 10:44 PM
grehan accepted this revision.Nov 11 2020, 11:03 AM
grehan added inline comments.
usr.sbin/bhyve/pci_emul.c
1115

It could be even simpler to just set pci_emul_membase64 to 3/4 * cpu_maxphysaddr

I should have been a bit clearer: the boundary doesn't have to be a power-of-2; just a multiple of a power-of-2, and that becomes the largest BAR size that could be placed. However, 3/4 of max phys addr is still an enormous number.

However, I'm fine with what you have.

This revision is now accepted and ready to land.Nov 11 2020, 11:03 AM
kib marked an inline comment as done.Nov 12 2020, 12:31 AM
kib added inline comments.
usr.sbin/bhyve/pci_emul.c
1115

I am working on the proper resource tracking for BAR allocation, in particular I already ported subr_vmem.c to userspace library. Having starting address be power of two is useful I believe, as the start of the resource region.

Closed by commit rS367606: bhyve: avoid allocating BARs above the end of supported physical addresses. (authored by kib). · Explain WhyNov 12 2020, 12:47 AM
This revision was automatically updated to reflect the committed changes.
kib marked an inline comment as done.
kib added a commit: rS367606: bhyve: avoid allocating BARs above the end of supported physical addresses..
kib added a commit: rS367607: bhyve: increase allowed size for 64bit BAR allocation below 4G from 32 to 128….Nov 12 2020, 12:52 AM
kib marked an inline comment as done.Nov 12 2020, 1:01 AM

Revision Contents
Changeset List

PathSize
usr.sbin/
bhyve/
29 lines

Diff 79243

View Options

usr.sbin/bhyve/pci_emul.c

Loading...