Select lacp egress ports based on NUMA domain

[gallatin]

Summary

This change creates an array of port maps indexed by numa domain for lacp port selection. If we have lacp interfaces in more than one domain, then we select the egress port by indexing into the numa port maps and picking a port on the appropriate numa domain.

This is behavior is controlled by the new ifconfig use_numa flag and net.link.lagg.use_numa sysctl/tunable (both modeled after the existing use_flowid), which default to enabled.

This is the first in a series of patches to use the numa information recently added to ifnet, mbuf and inpcb structs.

Test Plan

On a 4-node NUMA machine, attach NICs to domains 1 and 3 and use them in lacp. Use a dtrace script like this:
fbt::mlx5e_xmit:entry
/ args[1]->m_pkthdr.numa_domain == 3 /
{

dom = args[0]->if_numa_domain;
@dom = lquantize(dom, 0, 3, 1);

}

And observe that when use_numa is disabled, we see packets intended for domain 3 split evenly across the NICs on domain 1 and 3:

dtrace: script '/d/port.d' matched 1 probe
^C

value  ------------- Distribution ------------- count
    0 |                                         0
    1 |@@@@@@@@@@@@@@@@@@@@                     191481
    2 |                                         0
 >= 3 |@@@@@@@@@@@@@@@@@@@@                     183655

After enabling use_numa, we see these packets going out only domain 3

c368.sjc002.dev# /tmp/ifconfig lagg0 use_numa
c368.sjc002.dev# dtrace -s /d/port.d
dtrace: script '/d/port.d' matched 1 probe
^C

value  ------------- Distribution ------------- count
    2 |                                         0
 >= 3 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 297132

[markj]

sbin/ifconfig/ifconfig.8

2501

Style: new sentences should start on new lines.

Also, .Xr NUMA 4 ?

[gallatin]

Fix style issues pointed out by markj in the ifconfig man page change, and add cross-references to the numa(4) man page, as he suggests.

[• hselasky]

sys/net/ieee8023ad_lacp.c

1085

off by one??

if (domain >= MAXMEMDOM)

[gallatin]

Fix off-by-one error pointed out by hselasky

[gallatin]

sys/net/ieee8023ad_lacp.c

1085

Thanks for catching that!

[bz]

sbin/ifconfig/ifconfig.8

31

Please don't forget to update .Dd on the commit day

2516

Also lines should be wrapped properly at 72 or something; someone from manpages can help with the exact details.

sys/net/ieee8023ad_lacp.c

865

if you write it as

} else
#endif
{
   ..
}

you can save the extra #ifdef NUMA for the }

sys/net/if_lagg.c

271

Why do you set arg2 to 1 despite having arg1 not a NULL ptr?

[gallatin]

sbin/ifconfig/ifconfig.8

2516

Thanks for noticing. I managed to back out a patch that fixed this when I fixed something else. Sigh. I cannot wait until we can use a sane review system with pull requests.

[gallatin]

• Restored the fixes to the manpage formatting and xref's requested by markj (as noticed by bz)
• Updated the date on the manpage as requested by bz (speculative date for now)
• Changed ifdef / { matching as requested by bz
• Changed net.link.lagg.default_use_numa init as requested by bz (i think)
• Fixed printing of USE_NUMA flag in ifconfig, as noticed by me.

[gallatin]

sys/net/if_lagg.c

271

I'm afraid I cannot parse what you're asking for. Can you please elaborate? I think I may have fixed something, but I'm just guessing at this point.