ice: Add a failure injection facility

[kbowling]

Summary

Add compile-time optional, non-sleeping fail points around every VF
creation resource boundary, before VF VSI reconstruction, and in the
GET_STATS validation path.

Provide an ICE-wide wrapper and device selector so other driver
subsystems can add scoped points without duplicating the failpoint
plumbing.  Keep the current SR-IOV points and VF selector in an iov
child namespace.

Compile the facility only with options DRIVER_FAILPOINTS.  This shared
option avoids a separate kernel option for every driver that provides
test-only injection hooks.  Ordinary kernels contain no ICE failpoint
objects or sysctl nodes.  Require an exact PF device name and
optionally a VF index before any point can fire.  This prevents a stale
test setting from affecting another PF.

The hooks exposed two reset-lifetime defects while validating the
existing SR-IOV review series.  A PF reset could discard a firmware VSI
before teardown, and a rebuilt sibling could leave stale switch-filter
state after IOV destroy.

Tested on an E810-XXV with INVARIANTS and WITNESS.  All twelve creation
checkpoints rolled back and permitted immediate resource reuse.  Forced
reconstruction and malformed GET_STATS failures also preserved sibling
operation and reply cardinality.

MFC after:      2 weeks
Sponsored by:   BBOX.io

Test Plan

Can be triggered by hand or hooked into other test harnesses. The idea of keeping it behind a conf is to avoid polluting the sysctl tree for something only relevant to particular driver workers (as opposed to the iflib ones which are generic and could be hooked into CI on virtual devs etc). This is also to serve as a template for other driver workers.

[gallatin]

sys/conf/NOTES

445

I might make this more general. Eg, DRIVER_FAILPOINTS, so we don't have options explode for every driver.

[kbowling]

Accept @gallatin suggestion of general driver conf, add small targeted doc (will not enumerate fail points) on fail testing to ice(4)

[nick_spun.io]

In D58940#1353461, @kbowling wrote:

Accept @gallatin suggestion of general driver conf, add small targeted doc (will not enumerate fail points) on fail testing to ice(4)

I'm probably going to follow suit and add some fail points to aq(4) so this helps :)

[ziaee]

share/man/man4/ice.4

1167–1171

1167–1173

Is this a sysctl? Whatever it is, I'd like to have that context as shown. Also, see the suggestion for correct markup of kernel configuration declarations inside prose.

sys/dev/ice/ice_fault.h

2–31

We have updated the preferred license in the licensing policy, please take a look: https://docs.freebsd.org/en/articles/license-guide/

Suggestion: Note that SPDX short form is preferred, 2 clause is preferred over 3 clause, and "all rights reserved" is implied by the berne convention since 1989.

[ziaee]

Requesting changes for manual page markup.

[kbowling]

address license format and man page

[kbowling]

share/man/man4/ice.4

1167–1173

Sort of, but that's just the interface to a developer tool. See fail(9).

[ziaee]

share/man/man4/ice.4

1168

Nit: .Cd takes the rest of the line as an argument, so it does not need to be quoted

1169

Oh, okay thanks. Well if this fail is actually referring to a specific interface, maybe xreffing at the top of the section could make that more clear for those unfamiliar?

[kbowling]

address ice(4)

[ziaee]

Ok, yeah definitely this!