bhyve/amd64: Add support for CPUID configuration

[rosenfeld_grumpf.hope-2000.org]

Summary

Generate CPUID entries from any given CPUID config options and modify the emulated CPUID configuration for all VCPUs accordingly.

CPUID options conform to the following form:

-o [vcpu.<vcpuid>.]cpuid.<action>=<parameter>

When a CPUID option is rooted under 'cpuid.' in the config tree, it defines an entry for the global CPUID configuration, which is applied to all VCPUs. Any CPUID option rooted under 'vcpu.<vcpuid>.cpuid.' defines a CPUID configuration entry specifically for the given VCPU.

CPUID generation works in several phases for each VCPU:

• compile a list of global CPUID modification actions from the global 'cpuid' options (this is done only once)
• compile a list of per-VCPU CPUID modifications actions from the 'vcpu.<vcpuid>.cpuid' options
• build a CPUID configuration from the complete set of CPUID information emulated or passed through from the host CPU by VMM
• apply the global CPUID modification actions followed by the per-VCPU modification actions to the CPUID configuration
• perform a number of necessary checks and fixups on the modified CPUID configuration, like restoring APIC IDs, removing unreachable CPUID functions, checking for duplicates
• check the modified CPUID configuration against a predefined set of constraints for valid register contents for particular functions and indexes, such as bits that are immutable, are clear-only, or which can only be numerically lowered
• upload the final CPUID configuration for this VCPU to the VM

The most basic form of CPUID configuration parameter is the register modification action:

-o [vcpu.<vcpuid>.]cpuid.<leaf>[,<index>][+]=<register-action-list>

<leaf> specifies the CPUID leaf (aka "CPUID function") to create an entry for, with an optional index specified by <index>. Leaf and index correspond the EAX and ECX input register values, respectively, when a CPUID instruction is emulated for the guest.

<register-action-list> is a comma-separated list of actions for modifying the register values returned by CPUID for the specified leaf and index. Appending to an existing CPUID configuration parameter will automatically insert the comma.

Register operations take the following form:

<reg><op><value>

<reg> specifies one of the registers EAX, EBX, ECX, or EDX. <op> is one of "=" (assignment), "&=" (bit masking), or "|=" (bit setting). <value> specifies the value to use in the operation. If <value> starts with "~", the value will be inverted before it is applied to the register. There can be multiple operations applying to the same registers, they will be executed in the order specified.

There are also a number of "special actions" to set certain CPUID values in a more straightforward way:

• cpuid.hypervisor: Hypervisor identification ("hyperv", "kvm", "vmware", "virtualbox", "xen", or a raw 12-character string)
• cpuid.vendor: CPU vendor ("intel", "amd", or raw 12-character string)
• cpuid.family: CPU family
• cpuid.model: CPU model
• cpuid.stepping: CPU stepping
• cpuid.brand: CPU brand string (48 characters, leaves 0x80000002-5)
• cpuid.pkgtype: the package type for AMD CPUs (only "am3" at this time)
• cpuid.level: maximum supported CPUID function in the 0x00000000 range
• cpuid.xlevel: maximum extended CPUID function in the 0x80000000 range
• cpuid.linear: no. of address space bits for linear (virtual) addresses
• cpuid.physical: no. of address space bits for physical addresses
• cpuid.features: list of features to be enabled or disabled
• cpuid.check: behaviour of the CPUID constraints checks ("none": perform no constraints checks, "warn": issue a warning for failing checks, "fail": fail VM startup if a check fails (default), "fix": modify the value to conform to the constraints (implies warn))
• cpuid.fallback-style: The fallback method for inexistant/unsupported CPUID leaves, can be "intel" or "amd". Intel CPUs return the values for the highest valid CPUID leaf, while AMD CPUs return zeros. This is set automatic based on CPU vendor, but it may be overridden manually.

There are three sets of pre-defined CPUID configurations:

• x86 architecture levels: x86-64-v4, x86-64-v3, x86-64-v2, x86-64-v1
• Intel CPU models: icelake, cascadelake, skylake, broadwell, haswell, ivybridge, sandybridge, westmere, nehalem, core2duo, pentium4, coreduo, pentium3, pentium2, pentiumpro, pentium-mmx, pentium, 486
• AMD CPU models: opteron-g5, opteron-g4, opteron-g3, opteron-g2, opteron-g1, athlon-64, athlon-xp, athlon, k6-3+, k6-3, k6-2, k6, k5, 486

A pre-defined CPUID configuration can be selected by adding the option "model=<config>" to the CPU configuration on the command line:

$ bhyve [...] -c 4,model=x86-64-v2 [...]

This would load x86-64-v2.cfg from /usr/share/bhyve/cpuid. Of course, the pre-defined CPUID configurations selected by this method can be further modified by using additional cpuid.<...> configuration options. It's also possible to create completely new CPUID configuration files and putting them into /usr/share/bhyve/cpuid.

[chuck]

I like the overall direction of this, but this has a substantial foot print to review. Is there some way to smoke test some of this? Something simple like checking that each .cfg file passes check=fail would be a good sanity check, but more checks are better :)

sys/amd64/vmm/vmm_cpuid.c

334

Help me understand what is happening here. I think this check says "if the host has 48 or fewer address bits, tell the guest it has 49 address bits". If so, how does this help the guest?

usr.sbin/bhyve/amd64/cpuid.c

80

I think there should be some trailing \0\0\0 here to ensure this is 12 bytes.

1599

What happens here if the config is cpuid.hypervisor="x"? Would random data after the string get copied to act[]?

1637

Similar to hypervisor, what happens here if the config is cpuid.vendor="y"? Would random data after the string get copied to act[]?

[wanpengqian_gmail.com]

A data point on the 0x80000008 question, since we looked at the same leaf: VM_MAXUSER_ADDRESS_LA48 is 2^47 and log2() here rounds up, so log2(VM_MAXUSER_ADDRESS_LA48 + 1) is 48 and the check reads "if the configured width is <= 48, report 49". vmm's vmspace ends at VM_MAXUSER_ADDRESS_LA48 (vm_mem_init(&vm->mem, 0, VM_MAXUSER_ADDRESS_LA48) in vmm.c), so nothing at or above 2^47 can be mapped for the guest anyway, and a width above the host's (46 on our Haswell-EP boxes) means guest PTE bits the hardware treats as reserved under EPT/NPT. Was the intent perhaps the opposite clamp, MIN(host width, 47)? That would also cover 52-bit hosts, where today's legacy path passes the host value through.

Separately: keelOS will want exactly this for a CPU baseline across hosts (live migration), so we are interested in seeing it land. If it helps with the smoke-test question, we can run each .cfg with check=fail on the hosts we have (Haswell-EP, Skylake-SP, Comet Lake; AMD Zen 4 later) and boot FreeBSD and Linux guests on the result -- let me know if that would be useful and which tree to test.