Reallocate pcpu area on the correct domain.

[jeff]

Summary

This patch simply allocates various per-cpu data-structures on the correct domain. The actual pcpu area is reallocated and we waste a page. Recovering the page is likely more costly than the benefit as it will increase the size of the page array.

[kib]

sys/amd64/amd64/mp_machdep.c

406 ↗

(On Diff #60707)

You are remapping a page in the kernel .bss section, I think that pmap_enter() works there only by a chance for now. I am sure that we do not want to support such use of pmap_enter().

Ideally we would use something like pmap_qenter(), but .bss might be mapped with superpage.

[jeff]

sys/amd64/amd64/mp_machdep.c

406 ↗

(On Diff #60707)

What might break it? Why do we not want to support this?

[kib]

sys/amd64/amd64/mp_machdep.c

406 ↗

(On Diff #60707)

For instance, vm_page_array elements backing pages of kernel text/data/bss were not initialized until very recently. We never faulted on kernel segments, and pmap_enter() was correspondingly never needed there.

[jeff]

sys/amd64/amd64/mp_machdep.c

406 ↗

(On Diff #60707)

Ok but I still don't see a reason not to do this. It is simple and it works. Anything I do will just be a more long winded replacement of the same code.

[kib]

sys/amd64/amd64/mp_machdep.c

406 ↗

(On Diff #60707)

I do not object (strongly), just pointing out that the code introduces a new requirement.

Also, this causes demotion of the superpage mapping the data (.data + .bss), I believe.

[jeff]

If there is no strong objection I will commit this tomorrow.

[gallatin]

sys/amd64/amd64/mp_machdep.c

470 ↗

(On Diff #60707)

I'm getting a panic here on AMD rome configured in non-numa mode. I think we need to avoid acpi_pxm_get_cpu_locality() when we're not on a numa machine. The following patch seems to fix it:

        /* start each AP */
-       for (cpu = 1; cpu < mp_ncpus; cpu++) {
+       for (domain = 0, cpu = 1; cpu < mp_ncpus; cpu++) {
                apic_id = cpu_apic_ids[cpu];
-               domain = acpi_pxm_get_cpu_locality(apic_id);
+#ifdef NUMA
+               if (vm_ndomains > 1)
+                       domain = acpi_pxm_get_cpu_locality(apic_id);
+#endif
                /* allocate and set up an idle stack data page */
                bootstacks[cpu] = (void *)kmem_malloc_domainset(