Page MenuHomeFreeBSD

make lacp's use_numa hashing aware of send tags

Authored by gallatin on Feb 23 2020, 9:27 PM.


Group Reviewers

When I did the use_numa support, I missed the fact that there is a separate hash function for send tag nic selection. So when use_numa is enabled, ktls offload does not work properly, as it dos not reliably allocate a send tag on the proper egress nic since different egress nics are selected for send-tag allocation and packet transmit. To fix this, this change:

  • refectors lacp_select_tx_port_by_hash() and lacp_select_tx_port() to make lacp_select_tx_port_by_hash() always called by lacp_select_tx_port()
  • pre-shifts flowids to convert them to hashes when calling lacp_select_tx_port_by_hash()
  • add a numa_domain field to if_snd_tag_alloc_params
  • plumb the numa domain into places where we allocate send tags

In testing with NIC TLS setup on a NUMA machine, I see thousands of output errors before the patch when enabling kern.ipc.tls.ifnet.permitted=1. After the change, I see no errors, and I see the NIC sysctl counters showing active TLS offload sessions.

Diff Detail

rS FreeBSD src repository
Lint Skipped
Unit Tests Skipped
Build Status
Buildable 29561

Event Timeline

gallatin created this revision.Feb 23 2020, 9:27 PM
hselasky accepted this revision.Feb 23 2020, 9:36 PM

Looks good.

jhb accepted this revision.Feb 24 2020, 8:07 PM
rrs accepted this revision.Mon, Mar 9, 1:36 PM
This revision is now accepted and ready to land.Mon, Mar 9, 1:36 PM
gallatin closed this revision.Mon, Mar 9, 1:47 PM

Committed as r358808. (however I forgot to tag the review in the commit message)