(umaperf 7/7) Flag day, XDOMAIN and FIRSTTOUCH are now folded in to options NUMA. Zones may specify an override but UMA will pick a default.

[jeff]

Summary

This is part of a series of patches intended to enable first-touch numa policies for UMA by default. It also reduces the cost of uma_zalloc/zfree by approximately 30% each in my tests.

UMA_ZONE_NUMA has confused people. It should be UMA_ZONE_FIRSTTOUCH and it should be the default policy. Some zones want to over-ride the default. So this change makes it so a zone with no policy specified gets FIRSTTOUCH and otherwise it gets the specified policy. This enables XDOMAIN on all NUMA builds. In my testing this is no slower and will increasingly become faster as more of the kernel behaves better WRT NUMA.

[markj]

I don't really see why it's called UMA_FIRSTTOUCH. A zone that honours the first-touch policy as implemented just always returns domain-local memory (assuming that the allocating thread doesn't get preempted and migrated in the slow path), so this is actually more strict than first-touch as that is implemented elsewhere in the VM.

sys/vm/vm_glue.c

476–477 ↗

(On Diff #65699)

Why does this need to be explicitly specified?

[jeff]

In D22831#500078, @markj wrote:

I don't really see why it's called UMA_FIRSTTOUCH. A zone that honours the first-touch policy as implemented just always returns domain-local memory (assuming that the allocating thread doesn't get preempted and migrated in the slow path), so this is actually more strict than first-touch as that is implemented elsewhere in the VM.

Do you have a different suggestion? I find this much less surprising than NUMA and fits with the classic definitions. It's just a question of prefer vs fixed. We could permit fallback to other domains if we really wanted and add an additional flag.

[jeff]

sys/vm/vm_glue.c

476–477 ↗

(On Diff #65699)

It does not. None really need it now. It would be nice to leave it on bucket. Then I could make a tunable/compile time override for the default which was my original intent. But I'm not sure it's necessary now.

[markj]

In D22831#500082, @jeff wrote:

In D22831#500078, @markj wrote:

I don't really see why it's called UMA_FIRSTTOUCH. A zone that honours the first-touch policy as implemented just always returns domain-local memory (assuming that the allocating thread doesn't get preempted and migrated in the slow path), so this is actually more strict than first-touch as that is implemented elsewhere in the VM.

Do you have a different suggestion? I find this much less surprising than NUMA and fits with the classic definitions. It's just a question of prefer vs fixed. We could permit fallback to other domains if we really wanted and add an additional flag.

I agree it's better than NUMA. You might call them UMA_ZONE_NUMA_LOCAL and UMA_ZONE_NUMA_ROUNDROBIN, though those are kind of long names. That at least makes it easier to see that the different policies are mutually exclusive.