(umaperf 5/7) Use per-domain keg locks.
ClosedPublic
Actions

Authored by jeff on Dec 15 2019, 11:39 PM.

Details

Reviewers

mav
markj
rlibby
glebius
gallatin

Commits

rS356349: Use per-domain keg locks. This provides both a lock and separate space

Summary

This is part of a series of patches intended to enable first-touch numa policies for UMA by default. It also reduces the cost of uma_zalloc/zfree by approximately 30% each in my tests.

This locks each domain in the keg independently. Most keg fields that were non-domain were already read-only. uk_pages and uk_free become ud_pages and ud_free which is slightly annoying but not problematic. This allows your capacity to drain buckets to scale up with the number of nodes.

Because hash uses the keg lock and for simplicity elsewhere we force all !NUMA domains to use keg domain 0. It would be possible to limit this impact only to hash zones.

Diff Detail

Repository

rS FreeBSD src repository - subversion

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

jeff created this revision.Dec 15 2019, 11:39 PM

Harbormaster completed remote builds in B28169: Diff 65697.Dec 15 2019, 11:39 PM

jeff edited the summary of this revision. (Show Details)Dec 16 2019, 12:10 AM

jeff added reviewers: mav, markj, rlibby, glebius, gallatin.

markj added inline comments.Dec 16 2019, 7:45 PM

lib/libmemstat/memstat_uma.c
459 ↗	(On Diff #65697)	kread() return positive error values.
476 ↗	(On Diff #65697)	kread() returns 0 upon success.
sys/vm/uma_core.c
1188 ↗	(On Diff #65697)	`n` isn't being incremented anywhere.
1932 ↗	(On Diff #65697)	This comment is kind of misleading, it sounds like you want to check whether OFFPAGE was specified at keg initialization time.
3240 ↗	(On Diff #65697)	This is not a new problem with this patch, but I don't see how we ever get out of this loop when M_NOWAIT is specified and we are not doing a round-robin allocation. I think I broke this in r339686. Assuming I am not missing something I will fix it shortly.

Review feedback. bugs, etc.

Harbormaster completed remote builds in B28286: Diff 65903.Dec 21 2019, 8:47 PM

Also store per-domain information for kegs in round-robin domains.

Harbormaster completed remote builds in B28294: Diff 65935.Dec 23 2019, 8:34 PM

markj added inline comments.Dec 23 2019, 8:37 PM

lib/libmemstat/memstat_uma.c
476 ↗	(On Diff #65697)	This is still wrong?

markj added inline comments.Dec 23 2019, 10:12 PM

sys/vm/uma_int.h
257 ↗	(On Diff #65935)	Why pad the lock when the domain structure itself is padded?

jeff added inline comments.Dec 23 2019, 10:23 PM

sys/vm/uma_int.h
257 ↗	(On Diff #65935)	spinners slow down the lock owner, increasing hold time. For crossdomain free there can be a lot of contention on the keg locks.

markj accepted this revision.Dec 23 2019, 11:03 PM

This revision is now accepted and ready to land.Dec 23 2019, 11:03 PM

Looks good but I do have a question about how uma_zone_reserve() is supposed to work now.

sys/vm/uma_core.c
1258–1259 ↗	(On Diff #65935)	Now it inserts the slab into the partial list.
2399–2400 ↗	(On Diff #65935)	s/%ud/%u/
3198–3199 ↗	(On Diff #65935)	Just wanted to check that you intended how this changes the behavior for allocs with a reserve set. Now we keep the reserve on each of n domains. This should be fine as long as reserve are small. An alternative could be to distribute them through the domains ((howmany(reserve, ndomains) each), which I think should be fine as long as M_USE_RESERVE allocs are willing to try the other domains too.
1932 ↗	(On Diff #65697)	I think it's the flag that's misleading, more than the comment? I can take this as a note for flag cleanup: maybe these checks should be turned into UMA_ZONE_NOTPAGE (or whatever the spelling will be for the public API flag).

jeff added inline comments.Dec 24 2019, 8:58 AM

sys/vm/uma_core.c
3198–3199 ↗	(On Diff #65935)	Yeah the only user in tree is the vmem btag zone and the reserve is only set when there is no UMA_MD_SMALL_ALLOC. So it'll only be on non-numa machines really. There are something like 8 tags reserved per-cpu. So it's not a trivial amount of memory but in context it's not something I'm worried about.