ktls: Document requirement of use_flowid for lagg and NIC kTLS

[gallatin]

Summary

One non-obvious barrier to getting NIC kTLS to work on lagged interfaces is ensuring that lagg is running in use_flowid mode. If it is not, it will not allocate send tags, and software kTLS will be used.

[• hselasky]

sys/net/if_lagg.c

1906

I wonder if the NET_EPOCH_EXIT(et) should be last! You are reading some fields, and they may be freed!?

[gallatin]

Move NET_EPOCH_EXIT() to after the prints.

[gallatin]

sys/net/if_lagg.c

1906

I thought it was there to protect against the lagg port interfaces going away, but its easy enough to move it after the warning out of an abundance of caution.

[melifaro]

sys/net/if_lagg.c

1891

Maybe make warned another sc flag that gets removed/set on LAGG_OPT_USE_FLOWID flag change?
In that case we'd avoid having 'static' function variable and can write something like

if (__predict_false(sc->sc_opts & LAGG_NEED_WARN)) {
  NET_EPOCH_EXIT();
  warn_lagg_user(sc);
  return (EOPNOSUPP);
}

?