if_bnxt: add CAGR interrupt coalescing tick-resolution support

[sumit.saxena_broadcom.com]

Summary

Add the P7 CAGR (Completion AGgregation Ring) NQ-aggregation timer
tick-resolution knob, programmed via a direct GRC-window register
write, with a new dev.bnxt.<unit>.cagr_tick_res read-only tunable.
This is independent of the existing per-ring AGGINT_PARAMS coalescing.

[gallatin]

Does something need to re-apply this after reset?

Also, please run these through style(9).

sys/dev/bnxt/bnxt_en/if_bnxt.c

2067

Does this really need inline?

2078

Is bar=1 correct? Doesn't that land in the hwrm range? Convention seems to be 0 or 2.

2961

Does this override the tunable? If so, is that intended?

[sumit.saxena_broadcom.com]

sys/dev/bnxt/bnxt_en/if_bnxt.c

2067

No, we can drop inline.

2078

It should be HWRM bar (BAR 0). There are bunch of (ugly)issues here but collectively it works. Let me explain.

{readl/writel}_fbsd helpers are incorrect. When bar=0, then it writes to doorbell bar(BAR 2) and when bar !=0 , then it writes to HWRM bar (BAR 0). bnxt_set_cqcoal_tick_res() passes bar = 1, so it writes to HWRM bar (as bar != 0) so write lands to correct BAR and it works without issues. I will send a separate patch correcting {readl/writel}_fbsd APIs followed by v2 of this patch.

2961

It's incorrect, this bug was introduced while porting changes from out-of-box driver to upstream. It should be called before creating sysctls to set the default value.

[sumit.saxena_broadcom.com]

sys/dev/bnxt/bnxt_en/if_bnxt.c

2078

{readl/writel}_fbsd variants defined in bnxt_re driver are correct. {readl/writel)_fbsd defined if_bnxt.c are incorrect. As suggested in Rx timestamp support patch, we need to get rid off of these duplicates of {readl/write}_fbsd helpers.