iflib: Complete registration failure cleanup

[kbowling]

Summary

Pre-attach sysctls contain pointers into the iflib context. Any later
registration failure that frees the context must first remove that
sysctl tree.

Failures after a successful IFDI_ATTACH_PRE also did not consistently
call IFDI_DETACH or free the private taskqueue. In particular, routing
a taskqueue creation failure through the context cleanup could free the
driver softc while resources allocated by attach_pre remained live.

Track successful interrupt and queue setup and use one common unwind
path. Invoke IFDI_DETACH with IFNET_WLOCK dropped and release only
resources whose setup completed. Leave a failed IFDI_ATTACH_PRE to
unwind its own partial state, as required by the existing driver
contract.

A successful attach_pre can now be followed by detach before driver
queue allocation. Make the remaining queue-backed interrupt cleanup
paths tolerate absent queue arrays.

Mark a failed registration as detaching before draining the entire
private taskqueue. Drivers can register configuration tasks there, and
taskqueue_drain_all() does not wait for work queued during its drain.
Make every current non-admin callback reject detaching contexts so late
work cannot touch driver state.

Drain tasks and call ether_ifdetach() with neither the ifnet nor context
lock held. A callback already running may need either lock, while
ether_ifdetach() acquires ifnet_detach_sx. Reacquire IFNET_WLOCK before
the context lock to preserve the established lock order.

The shared automatic core-offset allocator also lacked acquisition
state. Late registration failures leaked its reference, while normal
detach could decrement a reference belonging to another device when a
configured offset or allocation failure meant that this context never
acquired one. Record acquisition explicitly and release only references
held.

MFC after:      2 weeks

Test Plan

https://reviews.freebsd.org/D58722

[kbowling]

Add one missed detach for SR-IOV

[gallatin]

sys/dev/bnxt/bnxt_en/if_bnxt.c

5504–5506

I really don't think this stuff needs to move if you're worried about the softc being in an inconsistent state. iflib_get_softc(ctx) just returns a pointer; it doesn't matter if its garbage until looked at. Same with cpr = &softc->def_cp_ring; The only thing that has to move is the deref of cpr

[kbowling]

Remove unnecessary bnxt churn

[kgalazka]

sys/dev/enetc/if_enetc.c

462

Maybe add

sc->rx_queues = NULL;

as you do with sc->regs for consistency?

[kbowling]

sys/dev/enetc/if_enetc.c

462

it has a different lifecycle, unlike regs rx_queues is not freed by enetc_detach. iflib calls IFDI_QUEUES_FREE after IFDI_DETATCH which frees and nulls the array.