kmem_back() picks the domain from the address and, when that domain has
no memory, steps forward until it finds one that does:
domain = (addr >> KVA_QUANTUM_SHIFT) % vm_ndomains; while (VM_DOMAIN_EMPTY(domain)) domain++;
Nothing stops the walk at vm_ndomains. When the empty domain is the
last one, the loop runs through the unused tail of vm_dom[] (all empty)
and on into whatever follows vm_dom[MAXMEMDOM] in memory, until a word
there happens to look like a non-zero page count; kmem_back_domain()
then allocates from that garbage domain. Ordinary allocations never
get here, because the per-domain arenas and the domainset iterators
skip empty domains. Only the direct callers of kmem_back() do: kinst
(one trampoline per thread, allocated when the module loads) and
memguard.
How we found it: keelOS runs FreeBSD as the root partition of its own
hypervisor, which on a two-socket host gives the root memory from the
first socket only, so SRAT has two domains and vm_dom[1] has 0 pages.
"kldload dtraceall" (which pulls in kinst) panicked every time:
kinst_trampoline_alloc_locked -> kmem_back -> kmem_back_domain -> vm_page_alloc_domain_after -> vm_reserv_alloc_page: page fault at 0x70000010
while every other provider loaded fine. Stock FreeBSD reaches the same
state on any NUMA machine whose last node has no memory, or simply when
hw.physmem is set low enough to leave the last node empty; the test
plan reproduces it in QEMU that way.
Wrap around with "% vm_ndomains" instead. The loop still terminates:
at least one domain has memory.
Signed-off-by: Wanpeng Qian <wanpengqian@gmail.com>
Sponsored by: keelos.dev