Fix several redundancy test reliability & debuggability issues.
tests/sys/cddl/zfs/include/libtest.kshlib:
- Add a generic wait_for mechanism that takes a timeout, dt, and a
command+args to run until it returns true.
- Add an is_pool_state <pool> <state> command.
tests/sys/cddl/zfs/tests/redundancy/redundancy.kshlib:
- cleanup: Always log the verbose status of the pool. This is
cleaner than doing it in umpteen different error cases, and in any case we don't usually bother generating output for success.
- sync_pool: Don't check the pool state. Make this the
responsibility of the several callers, which need to check for different states.
- damage_devs: In the damage-all-even-labels case, repeat the effort
to damage every vdev until it goes UNAVAIL. Previously, sometimes the pool would sync its labels to the device after the damage had been done but before the pool found the damage itself.
- clear_errors: Wait for the pool to become healthy rather than
requiring the 'zpool clear' to have this effect; resilvering will not necessarily complete immediately; it's issued async from clear.
- remove_devs: After removing the device files, wait for each of
them to become UNAVAIL in the pool config.
Submitted by: Will
Sponsored by: Spectra Logic Corp