Add PHC register mapping, RTC initialization, refclk read with
FW-reset serialization, and sbinuptime-based calibration to convert
the raw hardware timestamp carried in RX completions into an mbuf
timestamp. Advertise IFCAP_HWRXTSTMP, configure the RX timestamp
filter in bnxt_init(), and drain the calibration callout on stop.
Check HWRM_FUNC_QCAPS_OUTPUT_FLAGS_PTP_SUPPORTED in
bnxt_hwrm_func_qcaps() so the capability bit PTP is gated on actually
gets set, without which PTP never armed on real hardware.
Details
Diff Detail
- Lint
Lint Skipped - Unit
Tests Skipped
Event Timeline
- In addition to the inline comments, please make the code conform to style(9)
- There do not seem to be licenses on the new files
Don't we need to re-calibrate after down/up? Eg, bnxt_stop() drains the callout and clears rx_ts_enabled, but never resets clbr_done or the clbr_points[]. On the next bnxt_init(), bnxt_reset_calibration_callout() sees clbr_done != 0 and takes the callout_reset_sbt_curcpu() branch instead of calibrating so there's a full second during which packets are interpolated against very old ref. points.
| sys/dev/bnxt/bnxt_en/bnxt.h | ||
|---|---|---|
| 1487 | Can you clean this up? There are multiple definitions of this (and readl) in bnxt_re/main/c and bnxt_en/if_bnxt.c | |
| sys/dev/bnxt/bnxt_en/bnxt_ptp.c | ||
| 38 | Does this need a htole64? | |
| 52 | Are you sure you want getnanotime() and not just nanotime? Note that getnanotime uses a cached less precise, but faster to obtain, time. While nanotime always queries the timecounter for a very accurate time. | |
| 86 | Why a spin mutex? That's only needed in a hw interrupt context that cannot sleep. Network drivers typically do not run in that context on freebsd and can use a normal mutx. | |
| 93 | Seems to be a bit of a misnomer, since it doesn't deal with mbufs. That would be done at the iflib level. Can you please also include me on the review that adds rx timestamp support to iflib? Unless I'm missing something, I don't think iflib supports this now | |
| 121 | Do we need to worry about wrapping here? | |
| 128 | If i've got my units correct, we can overflow after 4.3s, which could be triggered by a few missed callouts. Maybe bound hw_clocks against hw_clk_div | |
| 129 | Is there any way to differentiate an uncalibrated (which is what i think hw_clk_div == 0 means) clock from a valid timestamp of 0? | |
| 310 | Don't you need to initialize this? bnxt_hwrm_cmd_hdr_init only fills in req_type, cmpl_ring, target_id, resp_addr. Every other field is stack garbage. I'm worried you could misconfigure the mac by accident. | |
| sys/dev/bnxt/bnxt_en/if_bnxt.c | ||
| 2904 | This is kind of weird. If we fail, we're going to return rc, and the attach will fail. But since rc is not checked, all the INIT* stuff below runs. All other failure cases jump to failed. Please either do that, or re-set rc to 0 if failure here is not fatal. | |
| sys/dev/bnxt/bnxt_en/bnxt.h | ||
|---|---|---|
| 1487 | Sure, will clean up. | |
| sys/dev/bnxt/bnxt_en/bnxt_ptp.c | ||
| 38 | Yes, it needs htole64(), will fix up this. | |
| 52 | I think we can use nanotime(). | |
| 86 | bnxt_calibration_callout (which calls bnxt_refclk_read) can get scheduled from the LAPIC timer interrupt while the CPU is in the idle thread. panic: mtx_lock() by idle thread 0xff01000235a96bd0 on mutex 0xffff00fff01000235a96bd0 @ bnxt_ptp.c:119
acpi_cpu_idle() at acpi_cpu_idle+0x299/frame 0xfffffe006152adb0
KDB: enter: panic Stopped at kdb_enter+0x33: movq $0,0x117e392(%rip) | |
| 93 | Ok, will fix up this. | |
| 128 | Ack | |
| 310 | Ack, will fix up this. | |
| sys/dev/bnxt/bnxt_en/if_bnxt.c | ||
| 2904 | Ack, will fix up this. | |
| sys/dev/bnxt/bnxt_en/bnxt_ptp.c | ||
|---|---|---|
| 93 | Here is the iflib changes to support rx timestamp: https://reviews.freebsd.org/D58638 | |
| sys/dev/bnxt/bnxt_en/bnxt_ptp.c | ||
|---|---|---|
| 206 | There is only one caller for this. Maybe just call bnxt_ptp_get_current_time() directly | |
| 239 | This is redundant.. you are checking that above to get into this block | |
| 317 | is this re-registering the driver? is that intended? | |
| 342 | FreeBSD error returns should not be negative | |
| sys/dev/bnxt/bnxt_en/bnxt_ptp.h | ||
| 37–42 | Aren't these already in bnxt.h? | |
| 79 | i think this is unused | |
Review comments:
- bnxt_ptp_timer() had exactly one caller (bnxt_calibration_callout()) and did nothing but re-check for a NULL ptp_cfg that bnxt_ptp_get_current_time() already checks itself. Inline the call to bnxt_ptp_get_current_time() and remove the wrapper.
- bnxt_calibration_callout()'s "HW timestamp frozen" branch re-tested priv->clbr_done != 0 inside a block already entered on priv->clbr_done &&; the inner check was always true. Flattened.
- bnxt_ptp_init() called bnxt_hwrm_func_drv_rgtr(bp, NULL, 0, true) with its return value discarded. With async_only=true this only re-sends the *default* async-events bitmap (no PTP-specific bits are added, since bmap is NULL) - the exact same bitmap the initial driver registration at attach already sent - so the call has no effect on what firmware forwards. Compare bnxt_register_async_events() in bnxt_ulp.c, which passes an actual extra bitmap for this same call to register RoCE's additional events; PTP does no such thing here. Removed as a no-op.
- This file mixed Linux-style negative errno returns (-EINVAL, -ENODEV, -EIO, -ENOMEM in bnxt_map_ptp_regs(), bnxt_ptp_init_rtc(), __bnxt_refclk_read(), and bnxt_hwrm_ptp_qcfg()) with the positive FreeBSD-convention errno already used elsewhere in the same functions and file (e.g. hwrm_send_message()'s return, and every other bnxt_en source file). None of the current callers inspect the sign, but it's a trap for any future == ENODEV-style check and inconsistent with driver style(9). Changed all five to positive.
- bnxt_ptp.h redefined BNXT_GRCPF_REG_CHIMP_COMM, BNXT_GRCPF_REG_CHIMP_COMM_TRIGGER, BNXT_GRCPF_REG_WINDOW_BASE_OUT, BNXT_GRC_BASE_MASK, and BNXT_GRC_OFFSET_MASK with byte-for-byte identical values to bnxt.h, which bnxt_ptp.h already #includes. Removed the duplicates.
- struct bnxt_ptp_cfg's clbr_gen field was never read or written anywhere. Removed.
AI scan (pass 1):
- The RX timestamp path dereferenced softc->ptp_cfg solely under if (softc->rx_ts_enabled), relying on an invariant enforced only in a different file (rx_ts_enabled is set true in if_bnxt.c only after ptp_cfg is allocated). Guard the deref directly with && softc->ptp_cfg != NULL so this file doesn't depend on that invariant holding across future changes.
AI scan (Pass 2):
- bnxt_ptp_free() destroyed ptp_lock but never callout_drain()'d tstmp_clbr, relying entirely on bnxt_stop() always running first (it does today, but bnxt_ptp_free() shouldn't depend on that ordering silently). Move callout_init() from bnxt_init() into bnxt_ptp_init() (paired with mtx_init()) so ptp_lock being initialized reliably implies the callout is too, then callout_drain() it in bnxt_ptp_free() under that same guard, before destroying the lock the callout body takes.
- bnxt_hwrm_ptp_qcfg()'s no_ptp: label freed ptp directly instead of calling bnxt_ptp_free(), which was safe only because every failure reaching it today happens before mtx_init(). If bp->ptp_cfg is ever reached here already holding a fully-initialized struct from a prior call, the mutex/callout would leak. Use bnxt_ptp_free(bp).
- bnxt_unmap_ptp_regs() unconditionally cleared a GRC window register that bnxt_map_ptp_regs() never actually maps (it rejects any refclk register requiring one, returning EINVAL). Removed the function and its one call site; there's nothing to unmap.
- bnxt_calibration_callout()'s frozen-counter path invalidated a calibration slot with atomic_store_rel_int(&curr->gen_ts, 0), a bespoke sentinel outside the seqcount API, only made to work by bnxt_ptp_hwtstamp_to_ns()'s non-standard if (gen == 0) return 0 check. Switched to the standard seqcount "permanently odd" idiom: the writer leaves gen_ts odd (still via a plain store, not seqc_write_begin(), which critical_enter()s with no matching seqc_write_end() to critical_exit() again here); the reader uses seqc_read_any()+seqc_in_modify() instead of seqc_read(), which would otherwise spin forever on a permanently-odd value instead of the transient-writer case it's designed for.
- Removed BNXT_HI_TIMER_MASK64, BNXT_HI_TIMER_SHIFT, and (once the bnxt_unmap_ptp_regs() removal above made it dead too) BNXT_PTP_GRC_WIN_BASE / BNXT_PTP_GRC_WIN: all unused.
- Renamed clbr_done to clbr_ticks: it's an unbounded per-tick counter gating the stall check and the first-tick path in bnxt_reset_calibration_callout(), not a boolean, despite the name.
I just threw this at my AI, which thinks that timestamps will almost never be set:
the conversion rejects nearly every normal packet timestamp.
Calibration runs once per second and records two points:
previous sample latest sample packet arrives
hw_prev = 1000 hw_curr = 2000 hw_tstmp = 2100
t = 0s t = 1s t = 1.1s
The conversion computes:
hw_clocks = hw_tstmp - hw_prev; // 2100 - 1000 = 1100
hw_clk_div = hw_curr - hw_prev; // 2000 - 1000 = 1000
Then D58596?download=true:353 rejects the packet:
if (hw_clocks > hw_clk_div)
return (0);
Because 1100 > 1000, conversion returns zero. The RX path only sets M_TSTMP for nonzero results, so the packet receives no timestamp.
The fundamental timing problem is that the latest completed calibration window is always in the past:
accepted by current code
┌─────────────────────────────┐
────────┴─────────────────────────────┴────────────── time
hw_prev hw_curr packet
Packets are normally processed shortly after hw_curr, before the next calibration callout. Consequently, their hardware timestamps are outside that historical
window. Only an unusually delayed packet captured before hw_curr but processed afterward would pass.
The calibration formula is intended to derive a clock rate from the previous interval and extrapolate that rate into the current interval. Existing FreeBSD mlx5
and cxgbe implementations do this without the hw_clocks <= hw_clk_div restriction.
A suitable fix is to allow bounded forward extrapolation—for example, up to roughly one additional calibration interval—while preserving an absolute bound and
overflow-safe arithmetic:
if (hw_clocks > 2 * hw_clk_div)
return (0);
The exact bound should include the scheduling tolerance discussed in P2. The important point is that limiting timestamps to [hw_prev, hw_curr] is incompatible
with real-time RX processing.