NUMA policy enhancements

[jeff]

Summary

This adds three features to cpuset & domainsets:

1. a dedicated set for the kernel (#2) so that we can dictate kernel and user policy separately. This also fixed the ugly interrupt root code.
2. DOMAINSET_POLICY_INTERLEAVE which does something like raid stripes or round-robin with a stride width. This gives much better performance than strict round-robin on the workloads i've tested on. I expect that we can tweak the stride as we learn how this works. Since rr is used as a fallback method when other things don't work I have kept it as a distinct policy.
3. A sysctl api for experimenting with domainsets on kernel objects and making the domainset_create() api kernel public with error checking. This allows you to put in a domainset 'key' and get back a validated policy that you can actually reference and use.

This also includes some header clean-up and a bug fix to the FIRSTTOUCH policy which allows us to switch it on as the default.

[jeff]

sys/vm/vm_domainset.c

93 ↗

(On Diff #40715)

This should be divided by sizeof not mod. Fixed.

[kib]

Am I right that striding does not account for the initial stride offset of the first allocation ? I.e. if the first allocation comes not from the start of reservation, then stride might spread over two vm_reservs ?

sys/kern/kern_cpuset.c

1156 ↗

(On Diff #40715)

Use bool for once ?

1343 ↗

(On Diff #40715)

return ();

[jeff]

sys/vm/vm_domainset.c

85 ↗

(On Diff #40715)

Add color to pindex for superpages so the stride aligns.