iflib: Track queue datapath lifecycle

[kbowling]

Summary

Track whether iflib queue mappings may still be accessed by the
device.  Keep the state private to iflib and conservative: an unknown
or failed device must pass through IFDI_STOP() before mappings are
reused or released, while a device known to be stopped need not
receive another hardware stop.

Enter the starting state before IFDI_INIT(), publish running only
after receive buffers and framework state are ready, and stop hardware
if receive-buffer setup fails after driver initialization.

Do not initialize an administratively-down interface merely because
its MTU, capabilities, VLAN configuration, or media changed.  Preserve
successful retries for an administratively-up interface whose previous
initialization failed.

This state describes ownership of iflib datapath mappings only.  It
deliberately makes no claim about firmware queues, administrative DMA,
PCI power state, or whether a driver can safely elide a hardware reset.

MFC after:      2 weeks

[kbowling]

Some background on this change is perhaps worth sharing.. I was originally chasing the up/down/up dance on boot, seeing if I could improve boot speed by carefully driving initial ifnet status through a state machine. But the majority of the time is spent in PHY code so doing something careful in iflib only shaved off xx-xxx milliseconds. Therefore the initial investigation is more of a driver state machine problem, and ice(4) has a good example of doing link state well. This ends up closing a few gaps anyway so here is just the lifecycle relevant part, and it lends naturally to the PM state machine that is a child of this review.

[gallatin]

sys/net/iflib.c

4675–4679

Maybe a dumb question, but if you're tracking the state, why do you need this complexity? Eg, why can't you check ctx->ifc_datapath_state != IFLIB_DP_STOPPED?

[kbowling]

sys/net/iflib.c

4675–4679

Not dumb, I updated the iflib_stop path to use the state. But I'm not sure if you are referring to the restart bool overall?