bhyve: Add support for specifying VM NUMA configuration

[bnovkov]

Summary

This patch adds basic support for adding NUMA domains to a bhyve VM.

The user can define a NUMA domain using the -n flag, which expects a domain id, a CPU set, and memory size for each NUMA domain.
After parsing the node configurations, we use the interfaces added in previous patches to set the NUMA configuration for the virtual machine.
Afterwards, we use the configuration to build the ACPI Static Resource Affinity Table (SRAT) which is used to pass NUMA information to the guest.

Users can optionally configure domainset(9) allocation policies for each domain.
Since each NUMA domain is essentially a separate SYSMEM segment, we can parse user-provided domainset(9) policies and install them into the backing vm_object of the appropriate segment.

Test Plan

I've tested the patch by booting a 10GB FreeBSD VM with two domains: -n id=0,size=5G,cpus=0-4 -n id=1,size=5G,cpus=5-9.
I can confirm that FreeBSD detects the specified domains properly.

bojan@dev /u/h/bojan> sysctl vm.phys_segs
vm.phys_segs: 
SEGMENT 0:

start:     0x1000
end:       0xa0000
domain:    0
free list: 0xffffffff81c073f0
...
SEGMENT 10:

start:     0x180000000
end:       0x2b6124000
domain:    1
free list: 0xffffffff81c078d0
...

[bnovkov]

Update patch.

[bnovkov]

Update patch:

• bhyve now uses cpuset(1)'s domainset(9) policy parser, allowing users to configure domainset(9) allocation policies for vm_objects backing each NUMA domain
• Added two usage examples to the manpage

[bnovkov]

Rebase

[markj]

usr.sbin/bhyve/bhyve.8

279

To me this description is unclear. NUMA might refer to GPAs or HPAs. As I understand from the code, this option lets you:

• divide guest physical memory into emulated domains, described by the SRAT;
• map those emulated domains to NUMA policies for the host physical memory which backs them.

Moreover, I think this lets you configure one independent of the other. In other words, you could use this option to:

• configure a NUMA policy for all of guest memory while keeping a single emulated domain (which is mostly redundant because cpuset(1) can already provide that)
• configure emulated domains while retaining the default domainset policy for all of them (useful for testing purposes).

Is that more or less correct?

280

281

303

Stray newline.

usr.sbin/bhyve/bhyverun.c

254

This could be for (i = 0; i < VM_MAXMEMDOM; i++)

278