Page MenuHomeFreeBSD

add NUMA support to powerpc
ClosedPublic

Authored by jhibbits on Nov 9 2018, 9:46 PM.

Details

Summary

Perfunctory NUMA support:

  • associate CPU with domain
  • associate memory ranges with domain
  • identify domain for devices
  • limit device interrupt binding to appropriate domain
  • Additionally fixes a bug in the setting of Maxmem which led to only memory attached to the first socket being enabled for DMA

A pmap variant can opt in to numa support by by calling numa_mem_regions at the end of pmap_bootstrap - registering the
corresponding ranges with the VM.

Diff Detail

Repository
rS FreeBSD src repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

mmacy created this revision.Nov 9 2018, 9:46 PM
mmacy edited the summary of this revision. (Show Details)Nov 9 2018, 9:53 PM
markj added inline comments.Nov 9 2018, 9:55 PM
sys/powerpc/powerpc/mp_machdep.c
188 ↗(On Diff #50236)

Rather than being conditional on options NUMA, it should be conditional on vm_ndomains > 1. One can disable NUMA support at boot-time with a tunable, and this breaks that.

sys/powerpc/powerpc/platform.c
121 ↗(On Diff #50236)

Extra newline.

136 ↗(On Diff #50236)

It'd probably be nice to say which.

167 ↗(On Diff #50236)

The locality table is optional, you can just pass NULL.

sys/vm/vm_page.c
785 ↗(On Diff #50236)

This shouldn't be necessary. We shouldn't be registering zero-length segments at all - they impose a runtime overhead on every operation that walks the segment array, and the array size is fixed.

kib added inline comments.Nov 10 2018, 7:13 AM
sys/powerpc/ofw/ofw_machdep.c
261 ↗(On Diff #50236)

This looks like a for() loop.

sys/vm/vm_page.c
785 ↗(On Diff #50236)

They might appear as a consequence of sloppy phys_avail[] manipulations, FWIW. At least x86 startup does a lot of surgery on it for early allocations.

Might be the forthcoming KPI would solve this.

markj added inline comments.Nov 10 2018, 5:54 PM
sys/vm/vm_page.c
785 ↗(On Diff #50236)

I was thinking of the vm_phys_segs[] array rather than phys_avail[] when I wrote that comment.

Which KPI are you referring to?

kib added inline comments.Nov 10 2018, 9:51 PM
sys/vm/vm_page.c
785 ↗(On Diff #50236)

D17431 (not committable)

jhibbits edited reviewers, added: mmacy; removed: jhibbits.Apr 7 2019, 6:20 PM
jhibbits commandeered this revision.

Will take this to completion.

jhibbits edited the summary of this revision. (Show Details)Apr 7 2019, 6:22 PM
jhibbits updated this revision to Diff 55917.

Fix the CPU domain binding. Only cpu0's domain was being bound, for all CPUs. Apply NUMA to HPT.

I just ran a test building llvm with ninja (3701 jobs):

With NUMA (144 threads): 25:32
Without NUMA (144 threads): 31:30
With NUMA (cpuset to socket 0, 72 threads): 24:35

This proves that the numa patch provides a significant performance boost. There is likely heavy contention in the pmap layer slowing things down below the single-socket case. I plan to measure and see what I can fix in that regard.

This revision was not accepted when it landed; it landed in state Needs Review.Apr 13 2019, 4:03 AM
This revision was automatically updated to reflect the committed changes.