HomeFreeBSD

mlx5e: Immediately initialize TLS send tags

Description

mlx5e: Immediately initialize TLS send tags

Under massive connection thrashing (web server restarting), we see
long periods where the web server blocks when enabling ktls offload
when NIC ktls offload is enabled.

It turns out the driver uses a single-threaded linux work queue to
serialize the commands that must be sent to the nic to allocate and
free tls resources. When freeing sessions, this work is handled
asynchronously. However, when allocating sessions, the work is handled
synchronously and the driver waits for the work to complete before
returning. When under massive connection thrashing, the work queue is
first filled by TLS sessions closing. Then when new sessions arrive,
the web server enables kTLS and blocks while the tens or hundreds of
thousands of sessions closes queued up are processed by the NIC.

Rather than using the work queue to open a TLS session on the NIC,
switch to doing the open directly. This allows use to cut in front of
all those sessions that are waiting to close, and minimize the amount
of time the web server blocks. The risk is that the NIC may be out of
resources because it has not processed all of those session frees. So
if we fail to open a session directly, we fall back to using the work
queue.

Differential Revision: https://reviews.freebsd.org/D47260
Sponsored by: Netflix
Reviewed by: kib

Details

Provenance
gallatinAuthored on Oct 23 2024, 7:16 PM
Reviewer
kib
Differential Revision
D47260: mlx5e: Immediately initialize TLS send tags
Parents
rGf3dbef108212: Bump __FreeBSD_version for llvm 19.1.2 merge
Branches
Unknown
Tags
Unknown