Add more fine-grained kernel options for NUMA support.

[jhb]

Summary

Add more fine-grained kernel options for NUMA support.

VM_NUMA_ALLOC is used to enable use of domain-aware memory allocation in
the virtual memory system. DEVICE_NUMA is used to enable affinity
reporting for devices such as bus_get_domain().

Note that 'cpuset -gd' always works.

If we left vm_ndomains at 1 in the !VM_NUMA_ALLOC case this would probably
reduce the diff. However, I would probably need to do something else to
handle bootstrapping 'cpuset -gd'. Perhaps I can just make the non-NUMA
fallback check to see if the set is empty and ignore it if it is non-empty
(though that assumes that domain 0 will always have at least one valid
CPU). I could perhaps check all of the sets and bail if any of them are
non-empty. The only other issue is if userland code checks vm.ndomains
and if so what is it expecting it to be used for. If it expects it to mean
that the kernel supports NUMA allocation then we should probably be leaving
it at 1.

Test Plan

• booted kernels with MAXMEMDOM increased and various combinations of the NUMA options

[kib]

sys/conf/options

601

Do we need this in opt_global.h ? The stuff is localized in acpica and _some_ vm/ files.

sys/vm/vm_page.c

469

Why would it bad idea to initialize all configured domains ? Then several pieces below would not need #ifdef VM_NUMA_ALLOC, mostly in ddb.

The only not very desirable consequence I see is that ddb commands would output empty domains, but I could argue that this is what should be done since the structures are allocated.

sys/vm/vm_phys.c

529

IMO this asks for two separate function bodies.

[jhb]

sys/vm/vm_page.c

469

Well, MAXMEMDOM structures are allocated. Even when VM_NUMA_ALLOC is configured, vm_ndomains can be < MAXMEMDOM. It is probably simpler if I just fix vm_ndomains to only be non-zero if VM_NUMA_ALLOC is true. I need to fix cpuset to stop using vm_ndomains, but that isn't a huge issue.

[jhb]

• Don't use vm_ndomains in the cpuset code.
• Only set vm_ndomains if VM_NUMA_ALLOC.
• Use a separate function body.
• Simplify now that vm_ndomains == 1 in the !VM_NUMA_ALLOC case.
• Move the NUMA options out of opt_global.h.

[kib]

sys/vm/vm_phys.c

51

I think it would be simpler and cleaner to include sys/proc.h unconditionally.

[jhb]

Dummy comment to workaround phab's bug that you can't mark something as done as a standalone change.