Disallow zvol-backed ZFS pools

[asomers]

Summary

Using zvols as backing devices for ZFS pools is fraught with panics and
deadlocks. For example, attempting to online a missing device in the
presence of a zvol can cause a panic when vdev_geom tastes the zvol.
Better to completely disable vdev_geom from ever opening a zvol. The
solution relies on setting a thread-local variable during
vdev_geom_open, and returning EOPNOTSUPP during zvol_open if that
thread-local variable is set.

Remove the check for MUTEX_HELD(&zfsdev_state_lock) in zvol_open. Its
intent was to prevent a recursive mutex acquisition panic. However, the
new check for the thread-local variable also fixes that problem.

Also, fix a panic in vdev_geom_taste_orphan. For an unknown reason,
this function was set to panic. But it can occur that a device
disappears during tasting, and it causes no problems to ignore this
departure.

[delphij]

LGTM.

[asomers]

In D4986#105668, @delphij wrote:

LGTM.

I think you told me that you also patched TrueNAS to disallow zvol-backed pools. Is that true? What approach did you take? If you're approach was better, we should take it instead.

[delphij]

In D4986#105673, @asomers wrote:

I think you told me that you also patched TrueNAS to disallow zvol-backed pools. Is that true? What approach did you take? If you're approach was better, we should take it instead.

That's true but I like your approach better than mine (which was a brutal hack in my opinion), but just in case you want to take a look:

https://github.com/trueos/trueos/commit/bcf44b3c6615b8edf39c26e064410d5c2c8f633e

[asomers]

In D4986#105674, @delphij wrote:

In D4986#105673, @asomers wrote:

I think you told me that you also patched TrueNAS to disallow zvol-backed pools. Is that true? What approach did you take? If you're approach was better, we should take it instead.

That's true but I like your approach better than mine (which was a brutal hack in my opinion), but just in case you want to take a look:

https://github.com/trueos/trueos/commit/bcf44b3c6615b8edf39c26e064410d5c2c8f633e

Ahh, that's what I tried at first. But it doesn't work, as you've probably figured out, because other geom providers like glabel can be layered on top of zvol.