Swap in processes unless there's a global memory shortage.
ClosedPublic
Actions

Authored by markj on Sep 24 2018, 6:45 PM.

Details

Reviewers

alc
kib
jeff

Commits

rS339601: Swap in processes unless there's a global memory shortage.

Summary

This is a hack to ensure that we will swap in processes even if one
domain is depleted (belongs to vm_min_domains). Add a domain selection
policy to kstack objects to ensure that we will attempt allocations from
all domains during a swapin.

Test Plan

Peter is testing the patch.

Diff Detail

Repository

rS FreeBSD src repository - subversion

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

markj created this revision.Sep 24 2018, 6:45 PM

Harbormaster completed remote builds in B19780: Diff 48404.Sep 24 2018, 6:45 PM

markj edited the test plan for this revision. (Show Details)Sep 24 2018, 6:46 PM

markj added reviewers: alc, kib, jeff.

markj added inline comments.Sep 24 2018, 6:50 PM

sys/vm/vm_object.c
277 ↗	(On Diff #48404)	We currently don't appear to do any initialization of the domainset ref, so I think we're relying on it being zeroed upon import into the object zone.

jeff added inline comments.Sep 24 2018, 6:51 PM

sys/vm/vm_glue.c
395 ↗	(On Diff #48404)	So this is necessary because the thread policy may not allow a given domain but we want to be able to swap? Long term it would be nice if kstacks and user policy lined up but for now I'm fine with fixing the issue this way.

markj marked an inline comment as done.Sep 24 2018, 6:55 PM

markj added inline comments.

sys/vm/vm_glue.c
395 ↗	(On Diff #48404)	Exactly. We could instead compute the intersection of the domainsets for all threads in a process, and allow swap-in if that intersection is not a subset of vm_min_domains.

markj marked an inline comment as done.Sep 24 2018, 7:10 PM

markj added inline comments.

sys/vm/vm_glue.c
395 ↗	(On Diff #48404)	... it should also be noted that the kstack may be cached and reused after a thread exits; some more work would be needed to ensure that a thread's kstack is always allocated using the thread's domain selection policy.

kib accepted this revision.Sep 24 2018, 7:16 PM

This revision is now accepted and ready to land.Sep 24 2018, 7:16 PM

markj mentioned this in D17059: Enable options NUMA on amd64 GENERIC/MINIMAL:.Sep 24 2018, 7:30 PM

I'm testing this patch against r338924 on the same EYPC 7551P 32core system used for testing against D17059.

In D17304#369051, @crest_bultmann.eu wrote:

I'm testing this patch against r338924 on the same EYPC 7551P 32core system used for testing against D17059.

NUMA enabled system work with this patch, but it feels like they still swap processes a lot faster (probably as soon as one NUMA domain is under memory pressure).

In D17304#370435, @crest_bultmann.eu wrote:

In D17304#369051, @crest_bultmann.eu wrote:

I'm testing this patch against r338924 on the same EYPC 7551P 32core system used for testing against D17059.

NUMA enabled system work with this patch, but it feels like they still swap processes a lot faster (probably as soon as one NUMA domain is under memory pressure).

Yes, that's true. I will revisit this once the current round of bugs has been addressed.

alc added inline comments.Oct 4 2018, 3:08 PM

sys/vm/vm_glue.c
319–321 ↗	(On Diff #48404)	Doesn't this need to account for empty domains?

markj marked an inline comment as done.Oct 4 2018, 3:12 PM

markj added inline comments.

sys/vm/vm_glue.c
319–321 ↗	(On Diff #48404)	Indeed. I'm working on a patch set to address some of the NUMA lowmem issues, and have rewritten this part of the diff in the process. I'll be uploading the patches later today. Instead, I define a static array domainset_prefer[MAXMEMDOM], which gets initialized early during boot (after SRAT is parsed and before vm_page_startup() is called). These sets get added to cpuset_domains, and later during boot we update the global domainsets in this list to exclude empty domains.

markj marked an inline comment as done.Oct 4 2018, 4:13 PM

markj added inline comments.

sys/vm/vm_glue.c
319–321 ↗	(On Diff #48404)	err, we're not using domainset_prefer[] here, but another global domainset, domainset_roundrobin.

Use the global domainset defined in D17416.
Don't start at domain 0 for each stack.

This revision now requires review to proceed.Oct 4 2018, 8:20 PM

Harbormaster completed remote builds in B19981: Diff 48748.Oct 4 2018, 8:20 PM

kib accepted this revision.Oct 5 2018, 6:01 PM

This revision is now accepted and ready to land.Oct 5 2018, 6:01 PM

markj mentioned this in D17416: Initialize global domainsets during boot..Oct 8 2018, 4:57 PM

alc accepted this revision.Oct 18 2018, 4:08 PM

alc added inline comments.

sys/vm/vm_glue.c
379 ↗	(On Diff #48748)	This will change to `DOMAINSET_RR()`, yes?
sys/vm/vm_swapout.c
745 ↗	(On Diff #48748)	The word "global" by itself here doesn't clearly describe the desired condition. It has too many possible interpretations. I would suggest a more precise statement: * - there exists at least one domain that is not suffering from a shortage of free memory;