Parallel zfs mounting

[jack_gandi.net]

Summary

This patch implements parallel mounting for zfs filesystems. This was first implemented on Illumos and ported to ZOL. After asking on the zfs-devel mailing list it was suggested that I start with the ZOL patch because it would be easier to port.

Delphix has seen huge decrease in latency of the mount phase with this patch (see results here: https://github.com/openzfs/openzfs/commit/a3f0e2b569). Performance improvements measured at Gandi are on par with those at Delphix.

Mailing thread here: https://lists.freebsd.org/pipermail/zfs-devel/2019-February/000516.html

Porting Notes:

• the ZOL patch pretty much applies directly to FreeBSD (unlike the Illumos patch)
• Had to take some code from the Illumos patch for #ifdef Illumos specific code that was removed when ported to ZOL
• After the ZOL port, a fix for a race condition was amended, this is included. (https://github.com/zfsonlinux/zfs/commit/e63ac16d25fbe991a356489c86d4077567dfea21)

ZOL-commit: https://github.com/zfsonlinux/zfs/commit/a10d50f999511d304f910852c7825c70c9c9e303

[jack_gandi.net]

Some updates:

This feature is listed here: https://docs.google.com/spreadsheets/d/1CFapSYxA5QRFYy5k6ge3FutU7zbAWbaeGN2nKVXgxCI/edit#gid=0 as not yet implemented for FreeBSD.
As far as I understand, FreeBSD is planning to rebase on ZoL. If this is the case we will get this feature for free with ZoF, so depending on how soon the rebase will this patch may or may not be useful to merge.

[avg]

I haven't thoroughly examined the change, but the procedure you followed sounds good to me.
And if it works, then it is good enough for me.

Also, after this is committed it would make sense to do a "record only" merge from the illumos vendor area for the corresponding illumos commit.

[sef]

Seems generally fine to me, although I haven't looked at the ZoL side for comparison.