malloc: Refactor redzone and KASAN handling

[markj]

Summary

malloc_large() duplicates redzone and KASAN handling that is also
present in malloc() and malloc_domainset(). Refactor the
implementations to reduce this a bit.

Reported by: Alexander Sideropoulos <[e-mail]>

[rlibby]

sys/kern/kern_malloc.c

110–111

Now unused.

732

DOMAINSET_RR() -> ds seems to be a functional change. Should it be separated?

741–743

I get why you're deleting this, because you now have malloc_domainset_exec just call this, but API-wise we lose the requirement that no one call malloc_domainset directly with M_EXEC. Is that the intention?

759–763

This didn't use to happen for malloc_large(), and now it does. Over in malloc(), it still doesn't happen for malloc_large(). I'm not understanding the difference.

[markj]

sys/kern/kern_malloc.c

732

Oops, yes, that's a logically separate change.

741–743

The reason for disallowing M_EXEC in plain malloc() is to avoid the overhead of handling a rare case in a hot path, i.e., an extra branch on M_EXEC. That reasoning doesn't really apply to malloc_domainset(), at least in my opinion: it's generally only called for long-lived allocations, so this kind of micro-optimization doesn't make sense.

759–763

I can't remember offhand why discrepancy exists, and there's probably no good reason. In general we want to call kmsan_orig() as close as possible to the consumer, as the use of KMSAN_RET_ADDR helps identify the origin of the allocation. I'll think about it some more and fix it, probably in a separate patch.

[markj]

Address review feedback