if_bnxt: add HW-based TX completion coalescing

[sumit.saxena_broadcom.com]

Summary

When firmware advertises BNXT_FLAG_TX_COAL_CMPL, request coalesced
completions via cmpl_coal_cnt on TX ring alloc and decode the
resulting tx_cmpl_coal_t records using a free-running SQ consumer
index per ring, instead of one completion per BD. Add the
tx_hw_coal_cnt sysctl to tune the firmware coalescing count code.

[gallatin]

sys/dev/bnxt/bnxt_en/bnxt.h

612

This does not seem to be re-initialized like cons/raw_cons/v_bit/toggle is in bnxt_init()

Why is this 64 bits? It looks like this is capped at 24 bits at most by hardware/

Maybe put it next to db_ring_mask? Then it would fit into an alignment hole.

sys/dev/bnxt/bnxt_en/bnxt_txrx.c

287

What's the point of the atomic load / stores here?

293

Shouldn't you be doing the store *AFTER* checking for !clear? Eg, if we hit this with clear==false, won't that corrupt the ring by advancing sq_cons without actually consuming anything? Eg, mbufs there would be orphaned.

In fact, setting this can introduce a race between the iflib interrupt filter and the gtaskq, where they might both write this. Not writing this in the filter (clear==0) eliminates the race and the need for atomics.

[sumit.saxena_broadcom.com]

sys/dev/bnxt/bnxt_en/bnxt.h

612

Ack

sys/dev/bnxt/bnxt_en/bnxt_txrx.c

287

We don't need atomics here.

293

Agreed.