Outbound IPsec packets retain the flow ID of the inner traffic. Multi-queue
interfaces use this ID to select a transmit queue (iflib's QIDX()) or a member
port (lagg(4) with use_flowid). Consequently, packets belonging to the same
security association (SA) can be distributed across independently draining
queues.

ESP and AH sequence numbers are assigned before encryption, but the packets
can be transmitted out of order when different queues drain at different
rates. This can cause legitimate packets to fall outside the receiver's
anti-replay window and be dropped as possible replays. RFC 4303, Section
3.4.3, makes 32 packets the smallest window a receiver must support, and
strongSwan installs that minimum by default.

Assign each SA a flow ID derived from a Jenkins hash of its SPI, computed once
when the SA is created, from the SPI in host byte order so that the value is
the same on every architecture. Set this ID on outbound packets in
ipsec{4,6}_perform_request(), with M_HASHTYPE_OPAQUE_HASH, once the SA is
known and before ipsec_accel_output(): it then applies whether the SA is
processed in software (ESP, AH, and IPCOMP) or by a network interface that
implements it, since the offload path keeps the packet header and hands the
packet to the interface's own transmit queue selection, and in a bundle the
last SA decides. Whether the flow ID selects the queue is up to the consumer:
a driver with its own selection (iflib's isc_txq_select) may ignore it.

Hashing the SPI distributes SAs across transmit queues regardless of whether
the queue-selection mechanism uses low-order bits (iflib) or shifted
high-order bits (lagg(4) with flowid_shift). It also avoids the poor
distribution that consecutive manually assigned SPIs would produce if used
directly.

All packets belonging to an SA now select the same transmit queue,
eliminating reordering caused by independently draining queues. Reordering
before enqueue, including skew in asynchronous crypto completion, remains
possible and is addressed separately.

Binding an SA to one transmit queue also binds it to one port or path when
using lagg(4) with use_flowid or flow-ID-based multipath routing. This trades
parallelism within an SA for packet ordering. Scaling a tunnel across
multiple queues is instead possible with multiple SAs, each with its own
sequence-number space, as supported by RFC 9611's per-resource Child SAs or
by separate policies or interfaces.

The net.inet.ipsec.sa_flowid sysctl restores the previous inner-flow-based
selection.

MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")

Tests on a 4-core Atom gateway with igc(4), four transmit queues, and four
SAs demonstrate the effect. Out-of-sequence counts and displacement come
from a tcpdump capture at the receiver's interface: a packet is out of
sequence when its ESP sequence number is lower than one already captured
for its SA, and its displacement is that difference. Replay drops are the
receiver kernel's "possible replay" count, taken after decryption, where
the receiver's own asynchronous crypto completion reorders further, so
drops also occur at displacements below the window size.

• In 15-second floods of 60-byte UDP packets with many inner flows per SA, 39-43% of ESP packets previously arrived out of sequence. Maximum sequence-number displacement was 6-9 packets with inline crypto and 14-29 with async_crypto=1, causing 6-251 replay drops per flood with a 32-packet window.

• With SA-based flow IDs, only 1.5-2% of packets arrived out of sequence with inline crypto, with maximum displacement of 3-4 packets and no replay drops. With async_crypto=1, 3.4-4.2% arrived out of sequence and 3-31 replay drops remained, due to completion-path skew addressed separately.

• In four-tunnel iperf3 tests, with four TCP streams per tunnel and three paired rounds toggling the sysctl in place, receiver replay drops fell from up to 0.7% of ESP packets to zero in every tested configuration. With async_crypto=1, out-of-order ESP arrivals fell from 10.8% to 0.2%, and maximum displacement from 195 to 2 packets.

• TCP throughput with async_crypto=1 and a 32-packet replay window increased from 1.29 to 1.42 Gbit/s. Throughput did not change with inline crypto or a 1024-packet window. CPU usage per Gbit/s was unchanged on both gateways.

The hardware measurements were taken with an earlier form of the change that
hashed the SPI per packet in ipsec_process_done(); the posted form stores
the hash in the SA when it is created and sets it in
ipsec{4,6}_perform_request(). On that gateway each SA was alone on its port,
so the two forms differ only in which queue an SA lands on, not in whether
its packets share one.

A test, tests/sys/netipsec/tunnel/sa_flowid, sends eight inner flows, each
with a flow ID of its own, over each of four SAs and checks that a lagg(4)
with use_flowid places every flow of an SA on the port given by bit 16 of the
SA's flow ID, and that with the sysctl off the inner flows of each SA spread
over both ports as before. It passes on the posted form (2/2 cases) and fails
without the change, as does the IPv6 flow label test of D61196 on top of it
(6/6), and the tests/sys/netipsec/tunnel suite shows no new failures, in a
FreeBSD 16-CURRENT VM with 8 CPUs.

The offload path (ipsec_accel_output() and the interface's queue selection,
mlx5e_select_queue() for mlx5(4)) was verified by reading only: no
offload-capable interface was available to run it.