Add more fine-grained kernel options for NUMA support.
ClosedPublic
Actions

Authored by jhb on Mar 30 2016, 5:15 PM.

Details

Reviewers

kib
adrian

Commits

rS297748: Add more fine-grained kernel options for NUMA support.

Summary

Add more fine-grained kernel options for NUMA support.

VM_NUMA_ALLOC is used to enable use of domain-aware memory allocation in
the virtual memory system. DEVICE_NUMA is used to enable affinity
reporting for devices such as bus_get_domain().

Note that 'cpuset -gd' always works.

If we left vm_ndomains at 1 in the !VM_NUMA_ALLOC case this would probably
reduce the diff. However, I would probably need to do something else to
handle bootstrapping 'cpuset -gd'. Perhaps I can just make the non-NUMA
fallback check to see if the set is empty and ignore it if it is non-empty
(though that assumes that domain 0 will always have at least one valid
CPU). I could perhaps check all of the sets and bail if any of them are
non-empty. The only other issue is if userland code checks vm.ndomains
and if so what is it expecting it to be used for. If it expects it to mean
that the kernel supports NUMA allocation then we should probably be leaving
it at 1.

Test Plan

booted kernels with MAXMEMDOM increased and various combinations of the NUMA options

Diff Detail

Repository

rS FreeBSD src repository - subversion

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

jhb updated this revision to Diff 14741.Mar 30 2016, 5:15 PM

jhb retitled this revision from to Add more fine-grained kernel options for NUMA support..

jhb updated this object.

jhb edited the test plan for this revision. (Show Details)

jhb added reviewers: adrian, kib.

Herald added a subscriber: imp. · View Herald TranscriptMar 30 2016, 5:15 PM

kib added inline comments.Mar 31 2016, 6:27 AM

sys/conf/options
601 ↗	(On Diff #14741)	Do we need this in opt_global.h ? The stuff is localized in acpica and _some_ vm/ files.
sys/vm/vm_page.c
469 ↗	(On Diff #14741)	Why would it bad idea to initialize all configured domains ? Then several pieces below would not need #ifdef VM_NUMA_ALLOC, mostly in ddb. The only not very desirable consequence I see is that ddb commands would output empty domains, but I could argue that this is what should be done since the structures are allocated.
sys/vm/vm_phys.c
529 ↗	(On Diff #14741)	IMO this asks for two separate function bodies.

jhb marked an inline comment as done.Apr 1 2016, 4:58 PM

jhb added inline comments.

sys/vm/vm_page.c
469 ↗	(On Diff #14741)	Well, MAXMEMDOM structures are allocated. Even when VM_NUMA_ALLOC is configured, vm_ndomains can be < MAXMEMDOM. It is probably simpler if I just fix vm_ndomains to only be non-zero if VM_NUMA_ALLOC is true. I need to fix cpuset to stop using vm_ndomains, but that isn't a huge issue.