Fix various ZFS test suite issues, improving reliability of the tests.

This change actually affects several different bugs, and is not being broken
apart, primarily because they are intertwined and often spread across many
files. Descriptions below per bug.

Make ZFS tests involving phy cycling more deterministic

tests/sys/cddl/zfs/include/libsas.kshlib:
tests/sys/cddl/zfs/tests/zfsd/zfsd.kshlib:

• Replace blind sleeps with functions to check phy state that accept

	  timeout parameters.

tests/sys/cddl/zfs/tests/hotspare/hotspare_replace_003_neg.ksh:

• Ensure that PHYs are always reenabled in the cleanup routine, even

	  if destroying a pool fails.

tests/sys/cddl/zfs/tests/hotspare/hotspare.kshlib:

• Only rmdir $HOTSPARE_TMPDIR if specified AND it is a directory.

tests/sys/cddl/zfs/tests/sas_phy_thrash/sas_phy_thrash_001_pos.ksh:

• Update test to check for disk failing to return, since

	  find_disk_by_phy just returns the state without judgment.

tests/sys/cddl/zfs/tests/hotspare/hotspare_replace_003_neg.ksh:
tests/sys/cddl/zfs/tests/zfsd/zfsd.kshlib:
tests/sys/cddl/zfs/tests/zfsd/zfsd_replace_002_pos.ksh:

• Update test code to rescan disks after reenabling phys. Instead

	  of doing this for each phy, make the tests now enable all PHYs
	  they are going to, then initiate the rescan.

Handle partitions more carefully during ZFS tests

tests/sys/cddl/zfs/include/libtest.kshlib:

• set_partition(): Don't create the partition table when adding

	  partitions; this hides bugs.

• wipe_partition_table(): Re-create the GPT partition table after

	  destroying it.  Commands must succeed.

• cleanup_devices():
  • Perform a ZFS labelclear before wiping the partition table,

	    since labelclear may destroy the partition table.
	  - Only wipe the partition table if a device is a physical disk as
	    opposed to a slice, used in some tests that still want to have
	    the ZFS label cleared.

tests/sys/cddl/zfs/tests/clean_mirror/setup.ksh:
tests/sys/cddl/zfs/tests/grow_pool/setup.ksh:
tests/sys/cddl/zfs/tests/grow_replicas/setup.ksh:
tests/sys/cddl/zfs/tests/interop/setup.ksh:
tests/sys/cddl/zfs/tests/inuse/inuse_008_pos.ksh:
tests/sys/cddl/zfs/tests/inuse/inuse_009_pos.ksh:
tests/sys/cddl/zfs/tests/migration/setup.ksh:
tests/sys/cddl/zfs/tests/no_space/setup.ksh:
tests/sys/cddl/zfs/tests/scrub_mirror/setup.ksh:
tests/sys/cddl/zfs/tests/write_dirs/setup.ksh:

• Update tests to ensure that a startup condition is honored: the test's disk partitions are cleared.

ZFS tests need to check every scenario, not a random subset

tests/sys/cddl/zfs/tests/cli_root/zpool_import/zpool_import_004_pos.ksh:
tests/sys/cddl/zfs/tests/cli_root/zpool_import/zpool_import_005_pos.ksh:
tests/sys/cddl/zfs/tests/cli_root/zpool_import/zpool_import_006_pos.ksh:
tests/sys/cddl/zfs/tests/cli_root/zpool_import/zpool_import_007_pos.ksh:
tests/sys/cddl/zfs/tests/cli_root/zpool_import/zpool_import_008_pos.ksh:
tests/sys/cddl/zfs/tests/cli_root/zpool_import/zpool_import_missing_001_pos.ksh:
tests/sys/cddl/zfs/tests/cli_root/zpool_import/zpool_import_missing_002_pos.ksh:
tests/sys/cddl/zfs/tests/cli_root/zpool_import/zpool_import_rename_001_pos.ksh:

• Refactor several tests so that they check every scenario instead of picking them at random. This caused spurious test failures.

tests/sys/cddl/zfs/tests/cli_root/zpool_upgrade/zpool_upgrade.kshlib:
tests/sys/cddl/zfs/include/libtest.kshlib:

• Fix assumptions about how the cachefile is managed.

zfsd(8) tests should ensure that zfsd is running

tests/sys/cddl/zfs/include/libtest.kshlib:
tests/sys/cddl/zfs/tests/zfsd/zfsd_*.ksh:

• Add ensure_zfsd_running: Check to make sure the zfsd service is

	  running, and start it if possible.  If this is not possible, fail
	  the test as unsupported.  This is aimed at identifying when a
	  system might not be set up properly.

• Update tests to call ensure_zfsd_running.

Reduce code duplication in ZFS test suite

tests/sys/cddl/zfs/...

• Replace repeated patterns of checking command results and failing,

	  with use of log_must/log_mustnot.
        - Replace copy-pasted code in various places with subroutines.

Submitted by: Will
Sponored by: Spectra Logic Corp