ice: Add a failure injection facility

[kbowling]

Summary

Add compile-time optional, non-sleeping fail points around every VF
creation resource boundary, before VF VSI reconstruction, and in the
GET_STATS validation path.

Provide an ICE-wide wrapper and device selector so other driver
subsystems can add scoped points without duplicating the failpoint
plumbing.  Keep the current SR-IOV points and VF selector in an iov
child namespace.

Compile the facility only with options DRIVER_FAILPOINTS.  This shared
option avoids a separate kernel option for every driver that provides
test-only injection hooks.  Ordinary kernels contain no ICE failpoint
objects or sysctl nodes.  Require an exact PF device name and
optionally a VF index before any point can fire.  This prevents a stale
test setting from affecting another PF.

The hooks exposed two reset-lifetime defects while validating the
existing SR-IOV review series.  A PF reset could discard a firmware VSI
before teardown, and a rebuilt sibling could leave stale switch-filter
state after IOV destroy.

Tested on an E810-XXV with INVARIANTS and WITNESS.  All twelve creation
checkpoints rolled back and permitted immediate resource reuse.  Forced
reconstruction and malformed GET_STATS failures also preserved sibling
operation and reply cardinality.

MFC after:      2 weeks
Sponsored by:   BBOX.io

Test Plan

Can be triggered by hand or hooked into other test harnesses. The idea of keeping it behind a conf is to avoid polluting the sysctl tree for something only relevant to particular driver workers (as opposed to the iflib ones which are generic and could be hooked into CI on virtual devs etc). This is also to serve as a template for other driver workers.

[gallatin]

sys/conf/NOTES

442

I might make this more general. Eg, DRIVER_FAILPOINTS, so we don't have options explode for every driver.

[kbowling]

Accept @gallatin suggestion of general driver conf, add small targeted doc (will not enumerate fail points) on fail testing to ice(4)

[nick_spun.io]

In D58940#1353461, @kbowling wrote:

Accept @gallatin suggestion of general driver conf, add small targeted doc (will not enumerate fail points) on fail testing to ice(4)

I'm probably going to follow suit and add some fail points to aq(4) so this helps :)