I have added a new if_alloc_domain() and if_alloc_dev() and ported 2 example drivers to use this new interface. (Assuming this patch is accepted, I will follow on with patches to at least iflib(4) and mxge(4)).
When called with a domain on a NUMA machine, ifalloc_domain() will record the NUMA domain in the ifnet, and it will allocate the ifnet struct from memory which is local to the device's NUMA node.
Drivers can call if_alloc_dev() to automatically record the NUMA domain in the ifnet and allocate the ifnet from local memory.
The NUMA node is used in a follow on patch to enhance lagg lacp egress port selection based on NUMA domain. (in addition to another patch which records NUMA domain information in the inp).
I have intentionally avoided ifdef NUMA as much as possible; all code is out of the critical path (only run at alloc/free) and gracefully falls back to the old behavior when run on a non-NUMA kernel or with a NUMA kernel on non-NUMA hardware.
The new if_numa_domain field fits in an alignment pad in struct ifnet, and so does not alter the size of the structure.