Page MenuHomeFreeBSD

bhyve: Add support for specifying VM NUMA configuration
Needs ReviewPublic

Authored by bnovkov on Mar 30 2024, 4:33 PM.
Tags
None
Referenced Files
Unknown Object (File)
Thu, Oct 3, 5:11 PM
Unknown Object (File)
Thu, Oct 3, 2:33 PM
Unknown Object (File)
Thu, Oct 3, 10:13 AM
Unknown Object (File)
Mon, Sep 30, 12:16 AM
Unknown Object (File)
Sat, Sep 28, 12:13 AM
Unknown Object (File)
Mon, Sep 23, 8:00 PM
Unknown Object (File)
Sun, Sep 22, 10:56 AM
Unknown Object (File)
Wed, Sep 18, 3:35 AM
Subscribers

Details

Reviewers
jhb
markj
corvink
Group Reviewers
bhyve
Summary

This patch adds basic support for adding NUMA domains to a bhyve VM.

The user can define a NUMA domain using the -n flag, which expects a domain id, a CPU set, and memory size for each NUMA domain.
After parsing the node configurations, we use the interfaces added in previous patches to set the NUMA configuration for the virtual machine.
Afterwards, we use the configuration to build the ACPI Static Resource Affinity Table (SRAT) which is used to pass NUMA information to the guest.

Users can optionally configure domainset(9) allocation policies for each domain.
Since each NUMA domain is essentially a separate SYSMEM segment, we can parse user-provided domainset(9) policies and install them into the backing vm_object of the appropriate segment.

Test Plan

I've tested the patch by booting a 10GB FreeBSD VM with two domains: -n id=0,size=5G,cpus=0-4 -n id=1,size=5G,cpus=5-9.
I can confirm that FreeBSD detects the specified domains properly.

bojan@dev /u/h/bojan> sysctl vm.phys_segs
vm.phys_segs: 
SEGMENT 0:

start:     0x1000
end:       0xa0000
domain:    0
free list: 0xffffffff81c073f0
...
SEGMENT 10:

start:     0x180000000
end:       0x2b6124000
domain:    1
free list: 0xffffffff81c078d0
...

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Tests Skipped

Event Timeline

bnovkov edited the summary of this revision. (Show Details)
bnovkov edited the test plan for this revision. (Show Details)

Update patch.

bnovkov edited the summary of this revision. (Show Details)

Update patch:

  • bhyve now uses cpuset(1)'s domainset(9) policy parser, allowing users to configure domainset(9) allocation policies for vm_objects backing each NUMA domain
  • Added two usage examples to the manpage