Add device bus support for NUMA domains; enumerate ACPI x86 device PXM entries

[adrian]

Summary

• Add a new device method that returns the VM domain ID of the given device, or -1 / ENOENT.
• Teach the x86 ACPI SRAT code to map PXM values back to VM domain IDs;
• Add methods to ACPI to perform a PXM lookup and map it to a VM domain ID;
• Print out the NUMA domain of a device (numa-domain X) if it exists.

A GENERIC kernel will just return -1 / ENOENT for all calls to bus_get_domain().

[adrian]

ugh, ignore the stuff in vm_phys.c.

[jhb]

sys/dev/acpica/acpi.c

1078

No "eventually" here, it does the mapping now. Also s/Fech/Fetch/. Also, I would say "_PXM" method instead of label because that is the nomenclature used in the ACPI specs. I would also possibly shorten the description a bit: "If a device has a _PXM method, map that to a NUMA domain." The rest of the comments just echo the code.

1091

Probably put ACPI_HANDLE above int and sort the int vars as 'int d, pxm'. This will save you a mail from Bruce later.

1097

You can combine the test from above here which would mean that bus_generic_get_domain() is only called via a fall-through if you do:

if (h != NULL &&
    ACPI_SUCCESS(...))) {
        ...
}

sys/dev/acpica/acpi_pci.c

350

As I suggested previously, you should make 'acpi_get_domain' non-static and re-use it here instead of duplicating.

sys/dev/acpica/acpivar.h

493

Perhaps "Map a _PXM value to a VM domain".

497

This line isn't needed as it is already obvious from the #if test on the next line.

sys/kern/subr_bus.c

3752

This is fine for testing, but shouldn't go into the commit. However, what I do think would be useful is to add a '%domain' sysctl node similar to '%desc', etc.

You could even only add that during device_add_child if bus_get_domain() is successful, and you can set it to the returned domain statically so the sysctl doesn't have to invoke BUS_GET_DOMAIN() each time it is called.

System administrators would much rather have a simple way ('sysctl dev.foo.0.%domain') to get this than having to do sed/awk magic on dmesg.boot.

sys/x86/acpica/srat.c

374

s/PXM/_PXM/

[adrian]

Yup, I'll do the sysctl stuff in a follow-up commit. It's a good idea; there's just a bunch of places i think we should expose the VM domain of CPU and CPU sets that it doesn't look like we currently are.

I'll fix this up and resubmit a patch soon.

Thanks!

[adrian]

Address jhb@'s concerns. Well, most of them.

[adrian]

Another update - don't hack in calling bus_print_child_domain() to bus_print_child_footer();
instead make it so the busses have to print the domain if it's relevant.

for now acpi and pci now print out the domain.

[jhb]

sys/dev/acpica/acpivar.h

498

Didn't see this earlier.. no need for extern for function prototypes. (We don't use it for other prototypes in the tree and in this file in particular.) You could also probably move these prototypes up into the earlier list (at least move acpi_get_domain up below acpi_MatchHid() perhaps).

sys/kern/subr_bus.c

3792

I think printing it for ACPI and PCI makes sense, but it's not clear to me it make sense to print this on i2c buses, etc. I may work on the sysctl node today btw.

[jhb]

sys/sys/bus.h

368

This could actually move up after bus_generic_get_dma_tag() as well.

[jhb]

You should also add a 'bus_get_domain()' wrapper (look for bus_get_dma_tag() in sys/bus.h and kern/subr_bus.c and just add it right after those). (I ran into this working on the %domain node.)

[adrian]

Cool, I'll commit those in a moment.

Thanks!

[adrian]

I committed a version of this to -HEAD.