HomeFreeBSD

ena: Fix false 'missing TX completions' warnings due to timestamp race

Description

ena: Fix false 'missing TX completions' warnings due to timestamp race

Sporadic 'Found a Tx that wasn't completed on time' warnings appear
under sustained TX load, always reporting '1 msecs since last cleanup'
despite the 5-second timeout threshold.

The per-packet TX timestamp uses struct bintime (128 bits: two 64-bit
fields sec and frac) which is read and written non-atomically. A race
exists between the missing TX completion check
(check_missing_comp_in_tx_queue reading the timestamp) and the TX
submit path or cleanup path writing it on another CPU. Since the two
fields are not updated atomically, the check can observe a partially
written timestamp - one field from the old value and one from the new.
This can produce a timestamp with {sec=0, frac=valid}, causing the
check to compute a time offset equal to system uptime and falsely
exceeding the 5-second timeout.

Confirmed by instrumentation showing all occurrences had sec=0 with
valid frac/mbuf, cleanup_running=0, and ticks==last_cleanup_ticks.

Replace struct bintime with sbintime_t (a single 64-bit value) for
tx_buf->timestamp. An aligned 64-bit store/load cannot be torn on
64-bit architectures. Additionally, snapshot the timestamp into a
local variable in the check path to prevent a read-then-read race
where the timestamp could be zeroed between the zero-check and the
offset calculation.

Testing:
On m6i.large (FreeBSD 15.0-RELEASE-p6 amd64, 2 IO queues), two
instances with MTU 1500. Ran iperf -P 20 -u -b 320kpps (CPU
saturated at ~7 Gbps aggregate).

Without the fix: 8 warnings in 6 hours (first at ~72 min).
With the fix: 0 warnings after 20+ hours under identical conditions.

Fixes: 9b8d05b8ac78 ("Add support for Amazon Elastic Network Adapter (ENA) NIC")
Submitted by: Gilad Ben Yakov <giladben@amazon.com>
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
Reviewed by: cperciva
Differential Revision: https://reviews.freebsd.org/D58241

(cherry picked from commit 74bcb1151bb94279a4269fc842aa2be00800545e)

Details

Provenance
Gilad Ben Yakov <giladben@amazon.com>Authored on Jun 21 2026, 12:45 PM
akiyano_amazon.comCommitted on Aug 1 2026, 3:12 PM
Reviewer
cperciva
Differential Revision
D58241: ena: Fix false 'missing TX completions' warnings due to timestamp race
Parents
rG203735fee058: ena: Batch RX statistics updates
Branches
Unknown
Tags
Unknown