Page MenuHomeFreeBSD

Swap in processes unless there's a global memory shortage.
ClosedPublic

Authored by markj on Sep 24 2018, 6:45 PM.

Details

Summary

This is a hack to ensure that we will swap in processes even if one
domain is depleted (belongs to vm_min_domains). Add a domain selection
policy to kstack objects to ensure that we will attempt allocations from
all domains during a swapin.

Test Plan

Peter is testing the patch.

Diff Detail

Repository
rS FreeBSD src repository - subversion
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

markj added reviewers: alc, kib, jeff.
sys/vm/vm_object.c
277 ↗(On Diff #48404)

We currently don't appear to do any initialization of the domainset ref, so I think we're relying on it being zeroed upon import into the object zone.

sys/vm/vm_glue.c
395 ↗(On Diff #48404)

So this is necessary because the thread policy may not allow a given domain but we want to be able to swap? Long term it would be nice if kstacks and user policy lined up but for now I'm fine with fixing the issue this way.

markj added inline comments.
sys/vm/vm_glue.c
395 ↗(On Diff #48404)

Exactly. We could instead compute the intersection of the domainsets for all threads in a process, and allow swap-in if that intersection is not a subset of vm_min_domains.

markj added inline comments.
sys/vm/vm_glue.c
395 ↗(On Diff #48404)

... it should also be noted that the kstack may be cached and reused after a thread exits; some more work would be needed to ensure that a thread's kstack is always allocated using the thread's domain selection policy.

This revision is now accepted and ready to land.Sep 24 2018, 7:16 PM

I'm testing this patch against r338924 on the same EYPC 7551P 32core system used for testing against D17059.

I'm testing this patch against r338924 on the same EYPC 7551P 32core system used for testing against D17059.

NUMA enabled system work with this patch, but it feels like they still swap processes a lot faster (probably as soon as one NUMA domain is under memory pressure).

I'm testing this patch against r338924 on the same EYPC 7551P 32core system used for testing against D17059.

NUMA enabled system work with this patch, but it feels like they still swap processes a lot faster (probably as soon as one NUMA domain is under memory pressure).

Yes, that's true. I will revisit this once the current round of bugs has been addressed.

sys/vm/vm_glue.c
319–321 ↗(On Diff #48404)

Doesn't this need to account for empty domains?

markj marked an inline comment as done.Oct 4 2018, 3:12 PM
markj added inline comments.
sys/vm/vm_glue.c
319–321 ↗(On Diff #48404)

Indeed. I'm working on a patch set to address some of the NUMA lowmem issues, and have rewritten this part of the diff in the process. I'll be uploading the patches later today.

Instead, I define a static array domainset_prefer[MAXMEMDOM], which gets initialized early during boot (after SRAT is parsed and before vm_page_startup() is called). These sets get added to cpuset_domains, and later during boot we update the global domainsets in this list to exclude empty domains.

markj marked an inline comment as done.Oct 4 2018, 4:13 PM
markj added inline comments.
sys/vm/vm_glue.c
319–321 ↗(On Diff #48404)

err, we're not using domainset_prefer[] here, but another global domainset, domainset_roundrobin.

  • Use the global domainset defined in D17416.
  • Don't start at domain 0 for each stack.
This revision now requires review to proceed.Oct 4 2018, 8:20 PM
This revision is now accepted and ready to land.Oct 5 2018, 6:01 PM
alc added inline comments.
sys/vm/vm_glue.c
379 ↗(On Diff #48748)

This will change to DOMAINSET_RR(), yes?

sys/vm/vm_swapout.c
745 ↗(On Diff #48748)

The word "global" by itself here doesn't clearly describe the desired condition. It has too many possible interpretations. I would suggest a more precise statement:

* - there exists at least one domain that is not suffering from a shortage of free memory;
markj added inline comments.
sys/vm/vm_glue.c
379 ↗(On Diff #48748)

Right, I just haven't re-uploaded the patch.

This revision was automatically updated to reflect the committed changes.
markj marked an inline comment as done.