Right now we have two flavours of vm_domainset iterators: "page" and
"malloc". The former is used by the page allocator, and the latter is
used by kmem_malloc() and friends. I would like to make the latter more
general.
The malloc iterator currently always derives its policy from
kernel_object, which has a DOMAINSET_POLICY_INTERLEAVE policy. However,
the iterator implementation just converts this to round-robin. Note
that the malloc iterator is used to select a vmem arena, which then
ensures that pages from the same 2MB virtual page are backed by physical
pages from the same domain. In particular, we're not relying on the
interleave policy to preserve superpages.
The malloc iterator currently also has the problem that it is run
without the kernel_object lock held, so updates to the iterator can race
with each other.
For the general case I would like to instead be able to specify a policy
in the vm_domainset_iter caller. In most cases this will just be a
round-robin policy, but this also permits the implementation of a
malloc_domainset(9) or uma_zalloc_domainset(9). This change refactors
the malloc iterator to support that case. I renamed it to be a "policy"
iterator, but better name suggestions are welcome. Policy iterators
always use curthread->td_domain.dr_iter as the iterator index, avoiding
races.
Aside from this, the lowmem handling for VM_ALLOC_WAITOK and M_WAITOK
allocations is not very robust. The current algorithm attempts NOWAIT
allocations from all domains in the set and then falls back to a WAITOK
allocation from the first domain in the set. If that domain's memory is
mostly wired, we may block indefinitely. Instead, we can use
vm_wait_doms() to sleep until _any_ domain in the set accumulates some
free pages. This change makes that adjustment.