A SO_REUSEPORT_LB group distributes incoming connections among its
members with a hash of the connection four-tuple. That hash is
unrelated to the receive queue the NIC steered the flow to, so a worker
pinned to the CPU that processes a given receive queue cannot be handed
the flows that land on its own CPU, and the locality that receive-side
scaling sets up is lost at the socket layer.
Add a SO_REUSEPORT_LB_CPU socket option that tags a member socket with a
receive-CPU affinity. When a packet is processed on that CPU,
in_pcblookup_lbgroup() delivers it to the matching member in preference
to the hash. Members without an affinity, and packets whose processing
CPU matches no affinitized member, keep the hashed distribution.
The group counts affinitized members, so the lookup scan is skipped for
groups that use none and the common path is unchanged. The match uses
curcpu at lookup time, the CPU the stack actually processes the flow on,
so it holds under both direct and deferred netisr dispatch and without
options RSS compiled in.