Fix logic in domainset_empty_vm().

[markj]

Summary

• Don't add 1 to the result of DOMAINSET_FLS(). It uses 1-indexing already.
• If a domainset consists entirely of empty domains, return true without modifying the set. Otherwise it is impossible to use DOMAINSET_FIXED reliably in kernel code.
• Always flatten a _PREFER policy to _ROUNDROBIN if the preferred domain is empty. Previously we were doing this only when ds_cnt > 1 after removing empty domains.

Test Plan

On a 2-socket system:

• I set hw.physmem to avoid using memory attached to domain 1, and verified that we can boot.
• I removed the DIMMs attached to socket 0, and verified that the system boots and that all system memory is associated with domain 1.

[kib]

Should there be a followup to restore _FIXED in amd64/mp_machdep.c after this commit ?

sys/kern/kern_cpuset.c

516 ↗

(On Diff #61300)

Is this DOMAINSET_NAND ?

[markj]

In D21420#466169, @kib wrote:

Should there be a followup to restore _FIXED in amd64/mp_machdep.c after this commit ?

I don't think so: the allocations must succeed, so if _FIXED is used there must be a fallback to RR if the allocation fails. But this is exactly the semantics provided by _PREF, so we might as well just use _PREF.

In general, _FIXED is dangerous since a M_WAITOK+_FIXED allocation will wait forever if the domain is empty. Any users should ensure that some upper layer specifies M_NOWAIT, as in uma allocf implementations.

[markj]

Use DOMAINSET_NAND.

[• hselasky]

This fixes boot issues with DELL PowerEdge R620 BIOS v2.7.0.

[alc]

head/sys/kern/kern_cpuset.c

479

Isn't this another instance of one of the mistakes that you fixed in domainset_empty_vm()?

[markj]

head/sys/kern/kern_cpuset.c

479

Thanks, I will fix it.